Training system for training a generator neural network

Abstract

A training system for training a generator neural network is provided. Some embodiments are directed to a training system (200) for training a generator neural network (100; 220), the generator neural network (100; 220) being arranged to transform measured sensor data into generated sensor data (223). The generator network is arranged to receive as input sensor data and a transformation target selected from a plurality of transformation targets, and to transform the sensor data in accordance with the transformation target.

Description

Technical Field

[0001] The present invention relates to a training system for training a generator neural network, a training system for training multiple individual generator neural networks, a generator system for a generator neural network, a training method for training a generator neural network, a training method for training multiple individual generator neural networks, a generator method for a generator neural network, a computer-readable medium including data representing instructions for causing a processor system to perform the method, and a computer-readable medium including data representing a neural network. Background Technology

[0002] Image-to-image translation is a class of vision and graphics problems in which the goal is to learn a mapping between an input image and an output image. Training can be done using a training set with aligned image pairs (called paired training data) or a training set without aligned image pairs (called unpaired training data). In the paper "Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks" by Jun-Yan Zhu et al., a known image-to-image translation system (called "CycleGAN") is described. CycleGAN learns to translate images from a domain... X Transform to domain Y And it does not require paired training data. Considering that they are in different domains... X and Y Any two unordered image sets X and Y Algorithms are known to learn how to transform images from one to another and vice versa.

[0003] The known system is the Generative Adversarial Network (GAN), which contains two mapping functions: G: X→Y and F: Y→X and two associated adversarial discriminators D Y and D X . D Y encourage G Will X Transform into domain Y Indistinguishable output, and for D X and F Conversely, the same applies. To further regularize the mapping, two cycle consistency losses are introduced. If an image is transformed from one domain to another and then returned, it should return to the original image. For the forward cycle consistency loss, the following is required: And for backward circular consistency loss: .

[0004] The models discussed above generate reasonable image-to-image conversion results. However, existing methods have several problems.

[0005] A drawback of known image-to-image translation systems—both those using paired training images and those not (such as CycleGAN)—is that they only learn transformations between two given domains in the training data and cannot perform any transformations beyond those two domains. Therefore, for multi-domain translations—transforming a given image into several target domains (e.g., a summer scene to an autumn, winter, or spring scene)—multiple generators and discriminators need to be trained, each dedicated to a specific domain transformation. Using multiple generators and discriminators for training is not only cumbersome and computationally expensive but can also lead to training instability. Balancing multiple generators and discriminators is difficult, and there is no guarantee that all transformation directions will be learned equally well.

[0006] Furthermore, in CycleGAN settings with two generators, it is typically observed that one generator dominates the transformation, while the other acts as a "helper." This limits the practical applicability of those models because their generalization and scalability properties are low.

[0007] While CycleGAN performs well in transforming images for some domains, the results are far from consistently positive. Performing high-quality transformations while preserving the meaningful semantics of a scene remains challenging. These methods are more likely to fail in more complex scenarios requiring greater variation and extreme transformations (especially geometric changes), or in scenarios with “new” unseen objects. For example, when transforming an image from winter to summer, one possible goal is to change snow to grass and adjust the background lighting. However, if the scene also contains foreground objects (e.g., people or animals), existing methods do not differentiate them from the background. The color and appearance of the foreground often change significantly during the transformation, and, for example, when transforming an image to the summer domain, one might give a wolf in a snowy landscape grass. The same problem is observed if the transformation goal is simply to change the foreground objects. For example, when transforming from an image of a horse to an image of a zebra, one might want to draw black and white stripes on the top of the horse while keeping everything else unchanged. However, in this example, the background color may also be affected during the transformation.

[0008] This behavior can be attributed at least in part to the fact that existing methods aim to learn meaningful transformations while looking at only two discrete points in space and time (two images, either paired or unpaired) during training. This is a very challenging task for networks, especially when trained with unpaired data. As a result, generators converge to solutions involving simple color and texture changes, without regard to the high-level semantics and plausibility of the synthesized scene.

[0009] By using synthetic images for data augmentation or for training other models, these drawbacks can lead to negative transfers, which in turn can result in decreased rather than improved performance of the other models. Summary of the Invention

[0010] To address these and other issues, a training system is proposed, which is configured to train a generator neural network that is arranged to transform measured sensor data into generated sensor data.

[0011] The training system includes a processor system configured to train a generator network together with a discriminator neural network. The generator network is configured to receive sensor data as input and a transformation target selected from a plurality of transformation targets, and is configured to transform the sensor data according to the transformation target. In an embodiment, a discriminator network is also configured to receive sensor data as input and a transformation target selected from the plurality of transformation targets, and to determine whether the sensor data is measured sensor data that satisfies the transformation target. The latter is preferred, but it is also possible to combine a single generator network with multiple discriminator networks, or vice versa.

[0012] The generator network and discriminator network receive sensor data to be transformed or discriminated, but also receive the transformation targets. In other words, the same generator network is trained to perform multiple transformations based on the multiple transformation targets. The same discriminator network is trained to discriminate between generated images and real images and / or between different transformation targets.

[0013] Therefore, training a different network for each transformation is avoided. The generator network of claim 1 is exposed to input and output image pairs according to multiple transformation targets. This improves network training. Furthermore, according to known systems, the size of a single network of claim 1 can be significantly smaller than two networks. G and FThe overall size is similar to that of a Cyclicgan generator network. In fact, implementations of generator networks can have a similar size to Cyclicgan generator networks. Using transformation objectives allows for an increase in the number of transformation objectives without a quadratic increase in the number of parameters. Moreover, training instability and imbalance are avoided by having fewer cooperating networks.

[0014] Therefore, a simpler yet more efficient framework is provided that enables multi-domain transformations using a single generator and discriminator, thereby reducing computational costs and improving the applicability of sensor data transformations in practice. During inference, a single generator can perform transformations to multiple domains and in different directions.

[0015] Improved training and / or an increased number of transformation targets allow the generator network to be applied in different settings. For example, the generator network can be trained to generate synthetic images for data augmentation or for training additional models.

[0016] There are at least two methods that can be used to transform the target. In an embodiment, the target of transformation can be a domain. A generator network can be trained to transform between different domains, and a discriminator network can be trained to discriminate between different domains. The number of domains can be two or more. For example, training data can be labeled using domains from multiple domains. For example, the label can indicate the domain. A domain can also be referred to as a class.

[0017] Sensor data can be labeled as categorical data (e.g., enumerated types), such as fields like summer and winter. Sensor data can also be labeled as quantitative data (e.g., continuous types), such as temperature and timestamps. Categorical labels can be used to define transformation objectives that result in a specific label. Quantitative labels can be used to define transformation objectives that increase or decrease the label by a specific amount.

[0018] Transformation objectives can also indicate different types of transformations that are not directly related to domain labels that can be associated with the image. For example, training images can be labeled using timestamps, and the transformation objective could be to transform an image to have a later or earlier timestamp. A particularly advantageous application of this is using training images from videos (e.g., time-lapse images) labeled with corresponding timestamps. Using videos is a particularly advantageous way to obtain large amounts of training data. Using transformation objectives, a generator network can be trained to age or youngen an image by a specific amount of time. Known generator networks are limited to transforming images between two seasons, but embodiments can age images by a specific number of days or hours, etc. Training objectives can be constrained in various ways, such as changing the time of day, changing the time of year, or both.

[0019] For image-to-image translation tasks, video data not only helps systems learn to recolor images but also to smoothly adjust the semantics of scenes while preserving high-level information and plausibility. Using video data is particularly beneficial because of its natural pairings. No human supervision is required to annotate labeled images. The spatial semantics of visual content evolves slowly over time, making it a valuable source of additional supervision.

[0020] Furthermore, the temporal distance between frames provides an additional source of information; two points that are close in time should appear similar, and they can be used to adjust the direction of transformation. In this context, the proposed framework can also be used to conjure up possible past or future scenarios conditioned on the current state.

[0021] Examples can be used on both paired and unpaired training data. Video is one way to obtain large amounts of paired training data. Video is attractive because it is cost-effective for large-scale acquisition while still containing rich signals. The temporal order of accompanying frames and strongly correlated nearby frames in a video is a valuable asset for supervision.

[0022] Multiple transformation targets can transform sensor data in at least two ways, although in embodiments this may be more than two, such as more than three, four, eight, etc., or even far more than two, such as more than 1,000, more than one million, etc. The targets can be general (e.g., transformations between seasons), but can also be quite specific, such as adding or removing a specific object at a specific location, such as adding or removing a cyclist at a specific location in an image, such as occluding or de-occluding an object, etc. Preferably, for each of the multiple transformation targets, there exists an opposite transformation target, such that performing two corresponding transformations sequentially should provide an identity for which an appropriate cyclic loss can be defined.

[0023] Such a multiple of transformation goals, which sequentially achieve identity, may be longer than two. For example, for three domains X, Y, and Z, such a loop of length two could be implemented, for example, X → Y → X or Z → X → Z However, there are also such loops with a length of three. X → Y → Z → X .

[0024] When time difference is used as a transformation objective, the opposite objective might be to delay or update sensor data by the same amount of time.

[0025] One aspect of the present invention relates to a generator system for a generator neural network, the generator neural network being arranged to transform measured sensor data into generated sensor data. Another aspect of the present invention relates to a training method for training a generator neural network. Another aspect of the present invention relates to a generator method for a generator neural network. Another aspect of the present invention relates to a transient or non-transitory computer-readable medium comprising data representing embodiments of a method to be performed by a processor system and / or data representing a generator neural network and / or a discriminator neural network according to an embodiment.

[0026] Generator networks and discriminator networks can also be referred to together as generative neural networks.

[0027] Training and generator systems are electronic devices, such as computers. A trained or untrained neural network can be represented by multiple parameters. One or more parameters can instruct how the neurons of the neural network transform input data into output data.

[0028] Sensor data can include any data received from a sensor. Sensor data can include input received from a sensor, for example, via a network, via an API, via a connection (e.g., a digital connection). The sensor can be an image sensor, radar, etc.

[0029] Sensor data can specifically include data extended in two or more dimensions. For example, sensor data can include planar data (such as image data), or it can be three-dimensional, such as including a depth component. For example, sensor data can have a time dimension; for example, sensor data can be sensor readings over a period of time. For example, sensor data can include video data. Sensor data can include, for example, video, radar data, LiDAR data, ultrasonic sensor data, motion sensor data, etc.

[0030] The generator neural network and / or discriminator neural network may include feedforward networks, such as frame- or video-based networks. The generator neural network and / or discriminator neural network may include CNN networks and / or RNN networks or architectures.

[0031] The trained generator network can be used to transform sensor data. In embodiments, image data is often used as an example of excitation. However, the network can be adapted to other modalities. For example, instead of visual images, radar data, etc., can be obtained.

[0032] An embodiment of a multi-domain transformation task utilizing a single generator and discriminator can be implemented between any sensor signals; time-series records from any sensor can be used as supervision. The embodiment can be used for data augmentation and domain transfer tasks. This is particularly valuable for training machine learning systems for autonomous driving. For example, by changing weather and lighting conditions, or shifting scenes to new environments (e.g., from urban to rural areas), and by transferring from synthetic data to real data, it enables realistic transformations of real images.

[0033] A particularly advantageous application of generator networks is to augment training or testing data for another network, such as a sensor data classifier, or an image classifier. Sensor data classifiers can be used in other devices, such as autonomously moving machines, like autonomous vehicles or robotic arms. In autonomously moving machines, sensor data classifiers can be used to classify objects near the machine. Based on the classified objects, a controller can be configured to generate control signals for controlling the machine (e.g., a physical system). The other device can be a computer-controlled machine, such as a robot, vehicle, household appliance, power tool, or manufacturing machine. Sensor data classifiers can also be used in systems for conveying information, such as surveillance systems, medical (imaging) systems, image annotation systems, etc. Sensor data classifiers can be used in personal assistants or access control systems, for example, to grant access to the system (e.g., via facial recognition).

[0034] In these other devices, neural networks (e.g., sensor data classifiers) typically operate well in a large number of cases; however, it is important to test such classifiers for extreme conditions. For example, autonomous vehicles can be tested on sensor data recorded in winter. In embodiments using generators, sensor data can be converted to summer data and used to test autonomous vehicles against summer data. For example, the machine can be used with sensor data recorded during the day. By using a generator, the sensor data can be made to reflect nighttime conditions. For example, it is known that machines perform poorly on sensor data that can be converted to a variety of different conditions and thus can be extended, for example, to different weather conditions, seasons, times of day, times of year, the number of other objects, visibility, etc. For example, it may be difficult to find neural networks (e.g., classifiers such as those used in autonomous machines or medical systems) that perform poorly on sensor data. Typically, these are very rare extreme cases. In an embodiment, the sensor data classifier (e.g., an image classifier) ​​is trained by: providing sensor data, such as sensor data on which the image classifier performs poorly, or sensor data on which the image classifier typically performs poorly; transforming the provided sensor data by applying a generator network with a transformation target, preferably multiple transformation targets, as in the embodiment, to obtain new sensor data, preferably multiple new sensor data; and using the new sensor data(s) for training the sensor data classifier.

[0035] Embodiments of the method can be implemented on a computer as a computer-implemented method, or on dedicated hardware, or a combination of both. Executable code for embodiments of the method can be stored on a computer program product. Examples of computer program products include memory devices, optical storage devices, integrated circuits, servers, online software, etc. Preferably, the computer program product includes non-transitory program code stored on a computer-readable medium for implementing embodiments of the method when the program product is executed on a computer.

[0036] In an embodiment, the computer program includes computer program code adapted to perform all or part of the steps of an embodiment of the method when the computer program is run on a computer. Preferably, the computer program is embodied on a computer-readable medium.

[0037] Another aspect of the invention provides a method for making the computer program available for download. This aspect is used when the computer program is uploaded to, for example, Apple's App Store, Google's Play Store, or Microsoft's Windows Store, and when the computer program is available for download from such stores. Attached Figure Description

[0038] Further details, aspects, and embodiments of the invention will be described by way of example only, with reference to the accompanying drawings. Elements in the figures are illustrated for simplicity and clarity and are not necessarily drawn to scale. In the figures, elements corresponding to those already described may have the same reference numerals. In the drawings,

[0039] Figure 1 An example of an embodiment of a generator network is illustrated schematically.

[0040] Figure 2a An example of an embodiment of the training system is illustrated schematically.

[0041] Figure 2b An example of an embodiment of the generator system is illustrated schematically.

[0042] Figure 2c An example of an embodiment of a generator network is illustrated schematically.

[0043] Figure 2d An example of an embodiment of a discriminator network is illustrated schematically.

[0044] Figure 2e An example of an embodiment of a discriminator network is illustrated schematically.

[0045] Figure 3a An example of an embodiment of the training set is illustrated schematically.

[0046] Figure 3b An example of an embodiment of the training set is illustrated schematically.

[0047] Figure 3c An example of an embodiment of the training set is illustrated schematically.

[0048] Figure 4a An example of an embodiment illustrating the details of a neural network is shown schematically.

[0049] Figure 4b An example of an embodiment of a neuron is illustrated schematically.

[0050] Figure 5a An example of an embodiment of sensor data is illustrated schematically.

[0051] Figure 5b An example of an embodiment of sensor data is illustrated schematically.

[0052] Figure 5c An example of an embodiment of sensor data is illustrated schematically.

[0053] Figure 6aAn example of an embodiment of the measured sensor data is illustrated schematically.

[0054] Figure 6b An example of an embodiment of the generated sensor data is illustrated schematically.

[0055] Figure 6c An example of an embodiment of a transformation target is illustrated schematically.

[0056] Figure 6d An example of an embodiment of the measured sensor data is illustrated schematically.

[0057] Figure 6e An example of an embodiment of a transformation target is illustrated schematically.

[0058] Figure 7a An example of an embodiment of the training method is illustrated schematically.

[0059] Figure 7b An example of an embodiment of the generator method is illustrated schematically.

[0060] Figure 8a A computer-readable medium having a writable portion including a computer program, according to an embodiment, is illustrated schematically.

[0061] Figure 8b A representation of a processor system according to an embodiment is shown schematically.

[0062] Reference number list:

[0063] 100 Generator Neural Network

[0064] 151 Sensor Data

[0065] 152, 153, 154 Change target

[0066] 120 Encoder Section

[0067] 130 Processing Section

[0068] 140 Decoder Section

[0069] 141 generated sensor data

[0070] 200 training systems

[0071] 210 Optimizer

[0072] 220 generator units

[0073] 230 Discriminator Unit

[0074] 240 training set storage devices

[0075] 242 Change target storage device

[0076] 250 Generator System

[0077] 252 Input Units

[0078] 254 output units

[0079] 221 Sensor Data

[0080] 222 Change target

[0081] 223 generated sensor data

[0082] 231 Sensor Data

[0083] 232 Change target

[0084] 233 Discriminator Output

[0085] Sensor data 234, 235

[0086] 236. Changing the target

[0087] 237 Discriminator Output

[0088] 238, 239 Discriminator Units

[0089] Sensor data measured in 301-303

[0090] Domains 311-313

[0091] 321-323 Other domains

[0092] Sensor data measured in 331-333

[0093] 340 Time Direction

[0094] 341-343 Timestamps

[0095] Training set 351-353

[0096] 361 Time Difference

[0097] 441-443 Sensor Data Channels

[0098] 451 Change target channel

[0099] 461-464 Convolution Filters

[0100] 470 Pooling Layer

[0101] 480 Fully Connected Layer

[0102] 410 Sensor Data Input

[0103] 411-412 Sensor Data Items

[0104] 420 Transform target input

[0105] 421 Transformation of target item

[0106] 430 neurons

[0107] 431 neuron output

[0108] 510, 520 Fields of sensor data

[0109] Sensor data 511, 521

[0110] Sensor data generated by 523 and 511'

[0111] 512, 522 Change target

[0112] 530, 540 sensor data domains

[0113] Sensor data from 531 and 541

[0114] 610 Measured sensor data

[0115] Sensor data generated by 620

[0116] 611 Car

[0117] 612 The people who are obscured

[0118] 613 The person who was removed

[0119] 630 Change Target

[0120] 631 Remove the obscured portion

[0121] Data from 640 measured sensors

[0122] 650 Change Target

[0123] 651 Remove the obscured portion

[0124] 1000 computer-readable media

[0125] 1010 writable portion

[0126] 1020 Computer Program

[0127] 1110 (one or more) integrated circuits

[0128] 1120 Processing Unit

[0129] 1122 Memory

[0130] 1124 Application-Specific Integrated Circuit

[0131] 1126 Communication Components

[0132] 1130 Interconnection

[0133] 1140 processor system. Detailed Implementation

[0134] While the invention is permissible in many different forms of embodiments, one or more specific embodiments are shown in the accompanying drawings and will be described in detail herein. It is to be understood that this disclosure is to be considered as an example of the principles of the invention and is not intended to limit the invention to the specific embodiments shown and described.

[0135] In the following description, for the sake of understanding, the elements of the embodiments are described in operation. However, it will be apparent that the corresponding elements are arranged to perform the functions described as being performed by them.

[0136] Furthermore, the invention is not limited to these embodiments, and the invention exists in each novel feature or combination of features described herein or recited in mutually different dependent claims.

[0137] In one embodiment, the training system includes two networks playing a competitive game: a generator G and a discriminator D. The task of G is to transfer a given sample x to a desired domain z. Y In this context, the transition occurs in the following manner: it follows the pattern of z. Y The distribution of the real data in the domain z is the same. D's task is to make the domain z... Y The real sample y in the domain is distinguished from any other sample or generated sample in the other domain.

[0138] Both D and G can be modeled as deep neural networks, where G takes the source image x as input and z as input. Y The target domain information is encoded in the image, and D takes the target image and its corresponding domain as input. The encoded domain information can be provided as input, along with the image, to the G and D networks, or / and to intermediate layer features by augmenting them with one or more additional channels containing domain information. When building the training, new data samples can be constructed based on the following: images and domain information, specifically, the currently represented... real Class (x) data , z X ) and (y data , z Y ), and indicate counterfeiting Class pairs (x) gen , z X(y gen , z Y ) together with (x data , z Y (y data , z X (x) gen , z Y (y) gen , z X The following pair—(x) data , z Y (y) data , z X (x) gen , z Y (y gen , z X — This helps the discriminator distinguish whether a target image is in the correct domain. These pairs can be used in the domain loss. Then the objective function can be:

[0139] .

[0140] The implementation can utilize variations of domain loss and cycle consistency loss in its loss function. The implementation can use identity loss for the generator, so G will be able to perform identity mapping, thereby learning to preserve better efficient characteristics between inputs and outputs.

[0141] The above setup does not require paired data for training. When paired data is available, a reconstruction loss can be used to allow G to learn a direct mapping between two domains. One example where paired data is freely available is time series data (e.g., video data), which can be used to train a better-performing G.

[0142] Implementations can be trained in a standard GAN manner using regularization techniques available in the literature. During inference, in contrast to previous work, a generator can perform transitions between multiple domains given source samples and target domains.

[0143] Interestingly, the generator domain can be trained to apply two transformations simultaneously. For example, sensor data can be transformed into the intersection of two domains. In such cases, the transformation can be encoded using a mapping function (e.g., a mapping that maps a combination of transformation targets to unique values). Alternatively, this can be done using multiple transformation channels. The latter option is preferred.

[0144] A preferred embodiment uses a single generator network and a single discriminator network, each with the transformed target as additional input. Class labeling is not required either. For example, a single generator network with the transformed target as additional input can be used, trained together with the discriminator network for each transformed target. For example, in the case of two domains, such a system would have three networks. In this setup, even the discriminator network benefits because the generator network becomes more sophisticated. Similarly, multiple generator networks (one for each transformed target) and a single discriminator network with the transformed target as additional input can be used.

[0145] For both category labeling and quantization labeling, the embodiments described herein assume the use of a single generator network and a single discriminator network. However, it should be noted that such embodiments for category labeling can be modified to use a single generator with multiple discriminators, or a single discriminator and multiple generators, by replacing the discriminator or generator, for example, with a CyclicGAN discriminator or generator, respectively.

[0146] Compared to CyclicGAN, the generator part is improved by replacing two networks with a single network that has the transformation target as an additional input; and the discriminator part is improved by replacing two networks with a single network that has the transformation target as an additional input. Furthermore, if both the generator and discriminator parts are each replaced by a single network, each part is improved, due to the benefits of replacing two networks with one network, and also because the other part is improved.

[0147] In a preferred embodiment, both the generator network and the discriminator network are arranged to receive a transformation target as input: the generator network is arranged to perform a transformation based on the transformation target, and the discriminator network is arranged to verify whether the sensor data matches the transformation target and whether the transformation target is real. However, benefits are still gained if either the generator network or the discriminator network is arranged in this way. Loss functions appropriately adapted to the situation, as disclosed herein, can be used.

[0148] In particular, multiple discriminator networks can be used in conjunction with a single generator network. In this case, the multiple discriminator networks are dedicated, for example, each discriminator network is associated with a specific transformation target, while the single generator network receives the transformation target as input.

[0149] In a particularly interesting variant, multiple generator networks can be used with a single discriminator network. In this case, the multiple generator networks are specialized; for example, each generator network can be associated with a specific transformation target, while the single discriminator network receives the transformation target as input. The advantage of this variant is that the generator network is improved due to the improved discriminator, but despite this, a network that does not require additional input is obtained. In an embodiment, a loss function is introduced into the discriminator system, which forces the discriminator to learn domain information. For example, the system may include a term in the loss function. .

[0150] For example, in one embodiment, a training system for training multiple individual generator neural networks, each generator neural network being arranged to transform measured sensor data into generated sensor data, the system comprising...

[0151] - Communication interface for accessing the training set of measured sensor data.

[0152] - A processor system configured to train multiple generator networks along with a discriminator neural network, wherein the generator networks are optimized to transform measured sensor data into generated sensor data, and the discriminator networks are optimized to distinguish between the measured sensor data and the generated sensor data.

[0153] - Each of the plurality of generator networks is associated with a corresponding transformation target and is arranged to receive sensor data as input and transform the sensor data according to the transformation target, wherein

[0154] - The discriminator network is arranged to receive sensor data as input and a transformation target selected from the plurality of transformation targets, and to determine whether the sensor data is measured sensor data that satisfies the transformation target.

[0155] Implementations with multiple generator networks or multiple discriminator networks can be adapted to the features described for implementations with a single generator network and a single discriminator network.

[0156] Figure 2aAn example embodiment of the training system 200 is illustrated schematically. The training system 200 is configured to train a generator neural network arranged to transform measured sensor data into generated sensor data. For example, the system 200 may include a generator unit 220 configured to apply the generator neural network, and a discriminator unit 230 configured to apply a discriminator neural network. For example, generator unit 220 and / or discriminator unit 230 may include storage means for storing parameters of the respective neural networks. For example, generator unit 220 and / or discriminator unit 230 may be configured to receive network input, apply the input and parameters according to the type of neural network, and provide network results at the output.

[0157] System 200 includes an optimizer 210. The optimizer 210 is configured to train a generator network along with a discriminator neural network. The generator network is optimized to transform measured sensor data into generated sensor data, and the discriminator network is optimized to distinguish between the measured and generated sensor data. To train the two neural networks, the optimizer 210 has access to a training set, for example, stored in a training set storage device 240. This training set includes the measured sensor data. The sensor data may be image data (e.g., images), but may alternatively or additionally include a wide variety of data, such as radar data, ultrasonic sensor data, etc. In an embodiment, sensor data may be obtained from a sensor configured to generate two-dimensional data characterizing the sensor environment. This sensor may be used in a machine. In an embodiment, at least a portion or all of the sensor measurement results have domain information and / or sensor time information indicating the domain and / or time in which the conditions (e.g., environment or type of environment) when the sensor data was obtained are located.

[0158] Sensor data can be multiple (potentially different sensor modalities) correlated sensor data. For example, in the example of an autonomous vehicle, a sensor data item may include one or more of image, radar, and other sensor data, typically concurrent data recorded from multiple sensors. For example, system 200 may include a communication interface for accessing a training set. Sensor data can be measured, for example, as received from sensors, and is, for example, real or genuine; or sensor data can be generated, for example, as generated by a generator unit, and is, for example, fake.

[0159] Once the generator network has been sufficiently trained—for example, after convergence, after exhausting the training data, or after a predetermined number of training iterations—it can be used in applications without a corresponding discriminator network. For example, Figure 2bAn example embodiment of generator system 250 is schematically illustrated. Generator system 250 is configured to apply a generator neural network, such as a generator neural network trained by system 200, for example, the generator neural network of generator unit 220. Therefore, generator system 250 is arranged to transform measured sensor data into generated sensor data. System 250 may include an input unit 252 configured to receive the measured sensor data as input. Input unit 252 may also be used to receive sensor data that is not measured but generated. After transforming the received input sensor data, the generated output sensor data can be placed, for example, on an output 254 for transmission. For example, system 250 may include a communication interface for receiving and / or transmitting sensor data.

[0160] System 250 includes a generator system 250 configured to apply a trained generator network to received input measured sensor data. Typically, system 250 is configured to perform additional tasks. For example, system 250 may be configured to augment additional training data for another neural network (e.g., for a classifier). System 250 and system 200 may be the same system, or they may not be the same system. System 200 and / or 250 may be a single device or may include multiple devices.

[0161] Systems 200 and / or 250 can communicate with each other or with external storage devices, input devices, or output devices via a computer network. This computer network can be the Internet, an intranet, a LAN, a WLAN, etc. The system includes a connection interface arranged to communicate as needed within or outside the system. For example, the connection interface can include connectors, such as wired connectors (e.g., Ethernet connectors, optical connectors, etc.) or wireless connectors (e.g., antennas, such as Wi-Fi, 4G, or 5G antennas, etc.).

[0162] The execution of systems 200 and 250 is implemented in a processor system, such as one or more processor circuits, examples of which are shown herein. Figure 2a and 2b The diagram illustrates functional units that can be functional units of a processor system. For example, Figure 2a and 2b These diagrams can be used as blueprints for the possible functional organization of a processor system. The processor circuitry(s) separate from the unit(s) is not shown in these diagrams. For example, Figure 2a and 2bThe functional units shown can be implemented, in whole or in part, as computer instructions stored at systems 200 and 250 (e.g., in the electronic memory of systems 200 and 250) and can be executed by the microprocessors of systems 200 and 250. In a hybrid embodiment, the functional units are implemented partly in hardware (e.g., as a coprocessor, such as a neural network coprocessor) and partly in software stored and executed on systems 200 and 250. The parameters and / or training data of the network can be stored locally at systems 200 and 250 or can be stored in a cloud storage device.

[0163] Back Figure 2a Interestingly, the generator network and discriminator network are arranged to receive sensor data as input and a transformation target. For the generator network, the transformation target indicates the transformation that the generator network needs to perform on the input sensor data. For the discriminator network, the transformation target indicates what type of input sensor data should be; for example, for the discriminator network, the transformation target indicates that the input sensor data has the same type as the type that the generator network attempts to create given that transformation target.

[0164] A transformation target can be selected from multiple transformation targets. The transformation targets can be discrete domains (e.g., a finite number) or they can be continuous (e.g., a high number). For example, system 200 may include a transformation target storage device 242. For example, a generator network may be configured to receive input sensor data and a transformation target, and to transform the input sensor data according to the transformation target. A discriminator network may be arranged to receive the sensor data as input and the transformation target, and may be arranged to determine whether the sensor data is measured sensor data that satisfies the transformation target. Preferably, for each target, there exists an opposite target; however, even if not, most preferably, the targets allow for the definition of loops, and preferably, loops can be defined for each operator. Loops of length two are preferably used, but longer loops may also be used.

[0165] Similarly, system 250 can also use a transformation target. For example, input unit 252 can be configured to receive the measured sensor data and the transformation target as input. Generator unit 220 in system 250 can be configured to apply a trained generator network to the received input measured sensor data and transformation target. The generator network in system 250 is arranged to receive the sensor data and transformation target as input, and is arranged to transform the sensor data according to the transformation target.

[0166] Instead of using a transformation target, the system can alternatively use multiple generator networks and / or multiple discriminator networks. However, in performing sensor data transformations (e.g., image-to-image conversion), using a transformation target is simpler and still effective. Computational costs are reduced and applicability is improved by using a single generator. In particular, the size of the trained network can be reduced. A single discriminator improves the discriminator and therefore the generator.

[0167] The discriminator network in discriminator unit 230 may have a single bit as output, such as yes / no, or real / fake. Multiple outputs are also possible, such as a single bit output indicating whether an image or images are real, and a single bit output indicating whether an image conforms to a transformation target; for example, a single bit output indicating whether an image or images are real, and a single bit output indicating whether the image or images are real and conform to the transformation target.

[0168] If sensor inputs are generated, particularly by a generator network, the discriminator network can be trained to output "forged." If the sensor data is measured (real) but does not conform to the transformation objective, the discriminator network can be trained to output "forged." If the sensor inputs are measured (real) and conform to the transformation objective, the discriminator network can be trained to output "true" or "true." Therefore, the discriminator network is trained to determine the sensor data and transformation objective based on the training set.

[0169] Typically, the discriminator is trained together for all three targets, such that the trained discriminator outputs "yes" for training pairs (e.g., a pair of measured sensor data and the corresponding transformed target), but outputs "fake" for anything else (e.g., a pair of generated data and any transformed target, and a pair of measured data but with an incorrect transformed target).

[0170] Transformation targets can be used to provide a wide variety of information to the network. In the first example, a transformation target can include or may be a domain. For example, for a generator network, a domain can indicate that the transformation is directed towards the indicated domain. For example, for a discriminator network, the domain can indicate that the input sensor data should be authentic and originate from the indicated domain.

[0171] For example, multiple transformation targets can include multiple domains, training data can be labeled using domains from multiple domains, a generator network is configured to transform sensor data to a domain according to the transformation target, and a discriminator network is configured to determine whether the input sensor data satisfies the domain according to the transformation target. Figure 3a An example of an embodiment of training set 351 is illustrated schematically. Figure 3aThe image shown is training set 351. Training set 351 includes multiple measured sensor data sets (sensor data sets 301-303 are shown) and multiple corresponding domains (domains 311-313 are shown). For example, the sensor data can be images, and the domains can be summer or winter.

[0172] Sensor data can be associated with multiple domains. Figure 3b An example of an embodiment of training set 352 is illustrated schematically. Training set 351 includes multiple measured sensor data, with sensor data 301-303 shown. Each sensor data can be associated with multiple corresponding domains. Domains 311-313 and 321-323 are shown. For example, the sensor data can be images, and the domains can indicate the season (summer / winter or summer / winter / autumn / spring, etc.) and landscape type (e.g., urban / rural). By using training set 352, the generator network and / or discriminator network can be configured to receive a single domain (e.g., the domain "summer") to transform a winter image of a rural or urban area into a summer image, or multiple domains (e.g., summer and urban, etc.). The same applies to the discriminator network. Domains can also be referred to as labels. For example, an image can be labeled with the label "summer" to associate it with the domain "summer".

[0173] Training data can be expanded using unlabeled data. Unlabeled data can also be used to train the system—either in cycle consistency or as additional training data for the discriminator. In the latter case, the discriminator's output can be changed to output two decisions: real or fake sensor data, and sensor data in or out of the domain.

[0174] For example, for additional consistency training, unlabeled images can be used for cycle consistency. For instance, unlabeled images can be mapped to a domain and then cycled, e.g., unlabeled -> X -> Y -> X'. Here, the generator network is applied three times, and the difference between X and X' should be small.

[0175] For example, you can use the label "real_image" which is satisfied by any image. The discriminator can then learn to distinguish between real images labeled "real_image" and generated images. In this embodiment, multiple outputs for the discriminator are particularly beneficial.

[0176] In one embodiment, the generator network and discriminator network are trained using only unpaired data. In this case, minimizing the cyclic loss can be used to ensure proper convergence of the network. In another embodiment, the generator network and discriminator network can be trained on paired data. In this case, pairs of sensor data representing the same situation but different only in terms of domain are known. For example, the same landscape, but in different seasons. The generator network and discriminator network can be trained only on paired data (e.g., obtained from video or time-lapse images). For example, the training set can include one or more sets of first measured sensor data, a transformation target, and second measured sensor data, and the generator network is trained to transform the first measured sensor data into the second measured sensor data according to the transformation target. Interestingly, the two networks can also be trained on a mixture of paired and unpaired training data. For the first case, a cyclic loss can be used; for the second type, a distance norm can be used to calculate the loss.

[0177] Using video or time-lapse images as training data has the added advantage of the slow, temporal evolution of the spatial semantics of visual content, which can be used as an additional source of supervision. For example, an additional regularization term can be introduced into the generator loss for minimization, thus constraining the generated images, or the features of generated images, to become more similar to each other as they become closer in time. In an embodiment, this can be achieved using the following mathematical formula:

[0178] ,in It can be any Lp norm (e.g., L1) or any other distance metric. The distance between the generated samples can also be computed in the feature space, for example:

[0179] ,in G l Indicate intermediate layer features. This allows additional free information from the video to be used as supervision for training.

[0180] It is important to note that the sensor data measured in different domains do not need to have the same sensor modality. For example, sensor modalities can be, for instance, audio and video data. This can be used to generate appropriate audio given a video, or appropriate video given audio. This can also be used in training data augmentation. For example, sensor modalities can be, for instance, radar and image data. This can be used to generate appropriate images given radar data, or appropriate radar data given image data. Image and / or radar data can be obtained from the vehicle and can be used to train autonomous vehicles. Different sensor modalities can be measured simultaneously to obtain paired training data. Different sensor modalities can be measured individually to obtain unpaired training data. Paired and unpaired data may be combined or not.

[0181] Figure 5a An example of an embodiment of sensor data is illustrated schematically. Figure 5a The diagram shows two domains of sensor data 510 and 520, for example, two domains. These two domains can be defined by corresponding images in the training data. For example, these two domains could be summer and winter, as defined by images in the training data labeled as summer or winter, respectively. For example, two sensor data sets, 511 and 521, are shown in domains 510 and 520, respectively.

[0182] Figure 5b An example of an embodiment of sensor data is illustrated schematically. Figure 5a The image shows measured sensor data 511. The measured sensor data 511 is transformed into generated sensor data 523 using a transformation target 522. For example, if the transformation target is a domain and the transformation target 522 is summer, then the sensor data 523 should be transformed to a summer setting. For example, if sensor data 511 represents sensor data acquired in winter (e.g., vehicle radar data and / or image data), then the generated sensor data 523 could represent vehicle radar data and / or image data acquired in summer. In this case, the opposite transformation represented by transformation target 512 is available. For example, transformation target 512 could correspond to the domain "winter". Applying a summer transformation to winter data 511 yields generated summer data 523, and applying a winter transformation yields additional generated sensor data 511'. The embodiment uses a so-called cyclic loss to optimize a neural network, according to which sensor data 511' should, for example, approximate sensor data 511 based on a distance norm.

[0183] The following is a more mathematical description of how the generator network G and the discriminator network D can be trained. Let X and Y be two domains (e.g., domains 510 and 520), and the goal is to learn the mapping G: X ↔ Y between unpaired or paired samples x ∈ X and y ∈ Y. The data distribution is labeled x ~ p data (x) and y~p data (y). Let y' (and y'') denote the transformed samples. The key requirement is that y' should appear to be extracted from the domain Y, while retaining the characteristics of sample x.

[0184] Domain information can be directly provided as input to the generator, i.e., y'=G(x, z) Y ), where z Y The target domain direction Y is encoded. This avoids the need for a second generator to learn the inverse transformation; in other words, we can make x' = G(y, z). X Given an appropriate loss function, both the generator and discriminator can be trained. Possible loss functions (e.g., the target function) that can be used are as follows:

[0185] ,in

[0186] .

[0187] In this paper, E denotes the expectation. Generally, the log-likelihood objective can be replaced by another scaling function (e.g., least squares loss). To ensure that the discriminator D learns to recognize that samples are generated in the target domain, an additional domain loss can be introduced. This additional domain loss penalizes the discriminator D if it does not see the target domain of the image correctly. For example, it can be achieved using:

[0188] .

[0189] The loss is particularly advantageous because it allows the discriminator to learn domain information. For example, it learns to penalize the generator if a generated sample is not in the correct domain. Without this loss, it would be much harder for the generator to produce samples in the correct domain. The first part is more important because it involves real samples used for the discriminator; in the embodiment, the parameter λ domain_gen It can be set to zero.

[0190] To learn domain mappings, implementations can leverage cycle consistency. For example,

[0191] (a) Forward cycle consistency loss: as well as

[0192] (b) Backward circular consistency loss:

[0193] These requirements can be expressed as

[0194] .

[0195] The objective could be to find a solution: Identity and reconstruction loss can be added to the training G:

[0196] Loss of identity: When provided with real samples from the target domain as input, identity loss encourages the generator to be near the identity map. This helps the map to preserve the color composition between the input and output.

[0197] Reconstruction losses:

[0198] In cases where paired samples are available.

[0199] Figure 5c An example of an embodiment of sensor data is illustrated schematically. (With) Figure 5a In comparison, the number of domains has increased to more than two; Figure 5c The diagram shows four domains. Besides... Figure 5a Beyond the territory, Figure 5c Sensor data fields 530 and 540 are also shown, for example, fields 530 and 540. Examples of sensor data in the fields are also shown (sensor data 531, 541).

[0200] By providing appropriate transformation targets, the generator network can transform sensor data to any of the domains. For example, sensor data 511, 531, and 541 can be transformed to domain 520. Having more than two domains increases the number of possible transformations. For two domains, there may be multiple transformation targets. For more than two domains, there may be more than two transformation targets among the multiple transformation targets.

[0201] To define cyclic loss, the transformation preferably allows for the definition of cycles. This is, for example, if it is possible to transform from any domain to any other domain.

[0202] The advantage is that the system can be easily extended to multi-domain transformations, featuring: a generator that performs transformations to multiple domains and in different directions; and a discriminator that evaluates whether a sample is real or generated and whether it is in the correct target domain. For example, the following can be used:

[0203] ,in

[0204] x i It comes from domain z i The sample consists of (i=1, ..., N) samples, where N is the number of target domains under consideration;

[0205]

[0206] In cases where paired samples are available.

[0207] The loss function above includes a loop of length two. This is preferred, but not strictly necessary; alternatives and / or extensions are possible. For example, if multiple transformation objectives include at least one multi-transformation objective, a cyclic loss can be defined such that sequentially transforming the measured sensor data according to the multi-transformation objective defines an identity transformation. For example, in Figure 5a In this scenario, the loop can go from field 510 to field 520 and then return. For example, for Figure 5c The loop can go from domain 510 to domain 520 to domain 540 and back to domain 510. The loss can include a loss term for this loop, for example, returning to domain 510 via domains 520 and 540 should be close to the original image.

[0208] Figure 2c An example embodiment of generator network 220 is illustrated schematically (e.g., it may be used in system 200 or 250). Generator network 220 is configured to receive sensor data 221 as input and a transformation target 222, and generate sensor data 223 as output. Sensor data 223 is, by definition, the generated sensor data. The input sensor data 221 may be (e.g., as available in the training set) measured sensor data, or it may be (e.g., used to calculate cyclic loss) the generated sensor data.

[0209] Figure 2d An example embodiment of the discriminator network 238 is illustrated schematically (e.g., as it may be used in system 200). The discriminator network 238 receives sensor data 231 as input and a transformation target 232. The discriminator network 238 can be trained to discriminate whether the input sensor data 231 is real (e.g., measured) and whether it corresponds to the transformation target 232. For example, if the transformation target is a domain, the discriminator network 238 can verify that the sensor data 231 is a member of that domain. The discriminator output 233 may be a binary output.

[0210] A discriminator with binary output works well; however, the discriminator's output can be varied. For example, in one embodiment, the discriminator network determines individually whether to generate an image or one of the input images, and whether that image is domain-dependent. Other options are possible.

[0211] The discriminator network 238 is particularly well-suited for labeled training data, for example, for transformations between domains.

[0212] Figure 2e An example embodiment of the discriminator network 239 is illustrated schematically (e.g., as it may be used in system 200). The discriminator network 239 receives sensor data 234 and sensor data 235 as input, and a transformation target 236. The discriminator network 239 can be trained to discriminate whether the input sensor data 234 and sensor data 235 are both real (e.g., measured) and whether they correspond to the transformation target 236, i.e., whether the change from sensor data 234 to 235 corresponds to the transformation indicated by the transformation target.

[0213] For example, when the target change is a time difference, the discriminator network 239 can verify that sensor data 235 differs from sensor data 234 by a time difference. For example, the time difference could be +4 hours, and the discriminator verifies whether sensor data 235 was measured +4 hours later than sensor data 234. Alternatively, the time difference could be -4 hours, and the discriminator verifies whether sensor data 235 was measured 4 hours earlier than sensor data 234. The discriminator output 237 can be a binary output.

[0214] If the training data is primarily paired, then the discriminator network 239 is particularly well-suited for training with stationary or time-lapse sensor data from videos. For example, sensor data acquired in the morning for an autonomous vehicle can be made to appear as if it were acquired in the afternoon. The discriminator network can learn whether images were acquired from the training set (e.g., both are real) and whether their difference has an indicative magnitude. One difficult case can be multiplied into many difficult cases.

[0215] Many types of neural networks are suitable for use in systems 200 and 250. For example, Figure 1 An example of an embodiment of a generator neural network 100 is illustrated schematically. The network 100 includes three parts: an encoder part 120, a processing part 130, and a decoder part 140.

[0216] Encoder section 120 is configured to receive input sensor data 151. Encoder section 120 may be configured to have a so-called bottleneck at its output. Processor section 130 receives the output of encoder section 120, and decoder section 140 may receive the output of the processing section. Transformation target 152 may be applied to one or more parts of the network. Figure 1 As shown, the transformation target 152 is provided as input 153 to the encoder section and as input 154 to the processing section. Although not shown in... Figure 1 As shown in the figure, it has been found that providing the transformation target as input to the decoder section 140 can be particularly advantageous.

[0217] In one embodiment, the transformation target can be the input to the decoder section 140. In another embodiment, the transformation target can be both the input to the decoder section 140 and the input to the encoder section 130.

[0218] In this embodiment, the encoder portion 120 includes multiple convolutional layers, the processor portion 130 includes multiple residual layers, and the decoder portion includes multiple convolutional layers. Various known types of layers can be added. For example, in this embodiment, the encoder portion 120 includes 5 convolutional layers, the processor portion 130 includes 4 residual layers, and the decoder portion includes 5 convolutional layers. The network can be larger or smaller as desired, or even much larger.

[0219] The discriminator network may include multiple convolutional layers, for example, five layers. At least one, but preferably more than one, layer receives the transformation target.

[0220] The generator network can be a known network; for example, for a generative network, it can be adapted according to J. Johnson et al.'s "Perceptual losses for real-time style transfer and super-resolution." However, additional inputs, including the transformation target, are provided to one or more of its layers. For the discriminator network, PatchGAN can be adapted, which aims to classify overlapping image patches (e.g., 70×70 patches) as real or fake. PatchGAN should also be adapted to receive additional inputs. According to embodiments, the adapted generator and discriminator networks can be trained using a loss function.

[0221] The network can be trained using any suitable solver, for example, the Adam solver with a batch size of 1; see D. Kingma and J. Ba. Adam, “A method for stochastic optimization”.

[0222] Figure 4a An example of an embodiment of a neural network is illustrated schematically. Figure 4a Multiple convolutional filters are shown (convolutional filters 461-464 are shown), and more or fewer filters may be present. Utilizing... Figure 4a The illustrated embodiment may be part of a generator network and / or a discriminator network. Figure 4a It can be, for example, the input to the first layer of the network, or it can be the input to the next layer. In the latter case, there can be additional inputs, for example, from the previous network layer.

[0223] The convolutional filter takes sensor data as input. For example, the sensor data may include multiple channels. Sensor data channels 441-443 are shown. For example, sensor data channels 441-443 may represent different aspects of the sensor data. For example, sensor data channels 441-443 may represent different color channels of an image, such as RGB channels. In this case, an additional channel is added: a transform target channel 451. The transform target encodes transform targets. For example, transform target channel 451 may encode two transform targets as a completely black or white plane (e.g., fully activated or deactivated). Multiple transform targets may be encoded in transform target channel 451, or they may be encoded by adding multiple transform target channels. For example, in this embodiment, each transform target corresponds to one channel.

[0224] In embodiments, multiple transformation targets can be encoded using a single transformation channel; for example, different values ​​in the transformation channel can correspond to different transformation targets. For example, if there are three transformation targets (e.g., three domains), the values ​​for the domains can be {0, 0.5, 1}. Preferably, the range of the input (e.g., sensor data input) (e.g., sensor data channel) is the same as the range of the input used to encode the multiple transformation targets. For example, each transformation target can be encoded as a discrete value, where the range of the discrete values ​​is the same as the range of the sensor data input. For example, the minimum and maximum values ​​are the same for sensor data input x and for transformation target z. It has been found that maintaining the same range improves optimization. For example, the ranges for sensor data input and for transformation target input can be the same interval, such as [0; 1]. Other transformation targets besides domains can be encoded with similar values. For example, time differences can be encoded as values ​​between 0 and 1, or as multiple values ​​between 0 and 1. For example, positive time differences can be encoded in a different channel than negative time differences.

[0225] Convolutional filters can straddle an input quantity defined by multiple channels. For example, a convolutional filter can receive a portion of each channel as input to compute a convolutional filter value in its output. By straddling the convolutional filter, multiple convolutional filter values ​​are obtained. The output of the convolutional filter can be further processed by the rest of the network. For example, in this example, a pooling layer 470 can be inserted after the convolutional layer, and a fully connected layer 480 can be placed after the pooling layer 470. These are merely illustrative, as many alternatives for network architectures are possible.

[0226] Convolutional filters 461-463 may be part of the first layer of the generator and / or discriminator network. Subsequent layers may also receive one or more input image channels, one or more transform target channels, and channels obtained from previous layers. For example, it has been shown that it is advantageous to provide one or more transform target channels to the decoding part of the generator network.

[0227] In this embodiment, the generator network and / or discriminator network are arranged to receive multiple channels as input, at least one of the channels encoding the sensor data and at least one of the channels encoding the transformed target.

[0228] However, it's important to note that sensor data input does not necessarily have to be 2D information, and may include additional information beyond 2D. For example, sensor input may include—potentially—audio, 1D radar data, etc., in addition to 2D information. Sensor data may also include 3D information, such as video or images with depth information for 3D. Sensor data may include sensor data with a time component, such as audio or video information.

[0229] Furthermore, it is not necessary to provide the transformation target as a channel. Although the latter is convenient, the transformation target can also be expressed as, for example, a bit vector.

[0230] Figure 4b An example of an embodiment of a neuron is illustrated schematically, which may be part of an embodiment of a generator network and / or a discriminator network. Figure 4bNeuron 430 is shown, which receives sensor data input 410 (e.g., data items obtained from the sensor data input) and transformation target input 420. For example, sensor data input 410 may include multiple sensor data items, shown as sensor data items 411-412. For example, a sensor data item may be a single bit (e.g., a 0 / 1 value), which, for example, represents a single bit of the sensor input (e.g., a pixel value, an input channel value, etc.). For example, transformation target input 420 may include one or more transformation target items, shown as transformation target item 421. For example, transformation target item 421 may represent a single bit (e.g., a 0 / 1 value), which, for example, represents a single bit of the transformation target. For example, transformation target item 421 may indicate a domain or time difference, etc.

[0231] In this embodiment, the transformation target corresponds one-to-one with the domain transformation. For example, sensor data can be labeled. The domain can be implicitly defined by sensor data with specific labels. The transformation can be defined as transforming an image so that it conforms to other images in the domain. In this example, the labels are discrete data, and the transformation is also discrete. However, the sensor data can be labeled alternatively or additionally using continuous labels. An example of continuous labels is a timestamp. In the case of continuous labels, the domain and transformation can still be defined as above. However, in this embodiment, the transformation target can be defined as the difference between two labels. For example, the label can be the external temperature. The transformation target can be to transform the sensor data to a specific external temperature, but alternatively, the transformation target can also be to transform the sensor data so that it corresponds to a higher or lower temperature.

[0232] For example, training data can be labeled using timestamps, and multiple transformation targets can include multiple time differences. The generator network can be configured to transform sensor data from a first timestamp to a second timestamp based on the time difference, and the discriminator network can be configured to receive first sensor data, second sensor data, and the time difference as input, and determine whether the first sensor data and the second sensor data satisfy the time difference.

[0233] Figure 3cAn example of an embodiment of training set 353 is illustrated schematically. Training set 353 includes measured sensor data, shown as measured sensor data 331-333. Associated with the measured sensor data are timestamps, shown as timestamps 341-343. The timestamps increment in the time direction 340. By using consecutive annotations, a generator network can be trained to generate sensor data based on increasing or decreasing timestamps. For example, given sensor data, the generator can generate an image with increasing timestamps (e.g., with an increase of time difference 361). Interestingly, in this case, it is known which image the system should generate, i.e., image 332. In other words, training data 353 can be used as pairwise training data. As a discriminator network, discriminator 239 can be used.

[0234] In an embodiment, sensor data (e.g., sensor data 331-333) may include images from time-lapse videos annotated with corresponding timestamps.

[0235] For example, the generator network in System 250 can be used to conjure possible past or future scenes conditioned on the current state. The generator G can be used to fill in training data. For example, missing images in a sensor data sequence can be filled in by delaying nearby images. Here, the generator G's job is to generate data from the current sample x. i and the expected time state Δt used for prediction i In the case of synthesizing possible future (or past) states, and the discriminator D needs to evaluate whether the sample is real or generated (x i+Δti or x i+Δti ') and two given samples x i and x i+Δti (or x) i+Δti Does it have a time distance Δt? i .

[0236] ,in

[0237] x i and x i+Δti It is a video with a time distance Δt i Two frames (i=1, ..., N), where N is the number of frames in the video sequence;

[0238]

[0239] Samples in a time series are paired by default;

[0240] Here, the identity loss is in Δt i Special reconstruction case when =0.

[0241] In various embodiments of systems 200 and 250, the communication interface can be selected from a variety of alternatives. For example, the interface may be a network interface to a local area network or wide area network (e.g., the Internet), a storage interface to an internal or external data storage device, a keyboard, an application interface (API), etc.

[0242] Systems 200 and 250 may have a user interface that may include known components such as one or more buttons, a keyboard, a display, a touchscreen, etc. The user interface may be configured to accommodate user interaction for configuring generator and discriminator networks, training networks on training sets, or applying the generator network to new sensor data.

[0243] Storage devices can be implemented as electronic memory (such as flash memory) or magnetic memory (such as hard disks), etc. A storage device can include multiple discrete memories that together constitute a storage device. A storage device can include temporary memory, such as RAM. A storage device can be a cloud storage device.

[0244] System 200 can be implemented in a single device. System 250 can be implemented in a single device. Typically, systems 200 and 250 each include a microprocessor that executes appropriate software stored at the system; for example, this software may have been downloaded and / or stored in a corresponding memory, such as volatile memory like RAM or non-volatile memory like flash memory. Alternatively, the system can be implemented entirely or partially as programmable logic, such as a field-programmable gate array (FPGA). The system can be implemented entirely or partially as a so-called application-specific integrated circuit (ASIC), such as an integrated circuit (IC) customized for its specific purpose. For example, the circuitry can be implemented in CMOS, for example, using a hardware description language such as Verilog, VHDL, etc. In particular, systems 200 and 250 may include circuitry for neural network evaluation.

[0245] The processor circuitry can be implemented in a distributed manner, for example, as multiple sub-processor circuits. The storage device can be distributed across multiple distributed sub-storage devices. Part or all of the memory can be electronic memory, magnetic memory, etc. For example, the storage device can have volatile and non-volatile components. Parts of the storage device can be read-only.

[0246] A particularly advantageous application is the extension of training data to additional neural networks, such as those different from the generator and discriminator networks. For example, this can be applied to data used for training or testing autonomous vehicles. For instance, system 200 can be trained on sensor data, including vehicle sensor data such as image data, radar data, etc. For example, the sensor data can represent the proximity of the vehicle, for example, as measured by one or more sensors included in the vehicle.

[0247] The target of the transformation can include time differences to generate sensor data based on different times of day and / or year. For example, sensor data acquired during a test run in the morning can be artificially delayed to appear as sensor data acquired in the afternoon, or vice versa.

[0248] The target of the transformation can include the vehicle's environmental type. For example, sensor data from an urban setting can be used for a rural setting.

[0249] The transformation target can include the type of sensor data. For example, image data can be transformed into radar data or vice versa.

[0250] The target of the transformation can include the color or model of the car. For example, a red car can be transformed into a blue car, or an SUV can be transformed into a hatchback, etc.

[0251] When training a system for autonomous vehicles, it's possible to use generated afternoon data in addition to morning data. The trained system could be, for example, an object classifier. Thus, the new network would be able to identify cyclists in the morning and afternoon, and in different landscapes. The expanded data can also be used to test autonomous vehicles in a laboratory setting. Many types of transformations are possible, and therefore, even large amounts of training data can be expanded to even larger amounts. These transformations can be applied to any sensor data, not just vehicle sensor data. The sensor data could be from another type of device (e.g., robotic arms, power tools, etc.).

[0252] Synthetic datasets (e.g., simulation engines) can be obtained, generating any amount of training data in the desired target domain. This can include the synthesis of so-called extreme cases, which are unlikely but important scenarios for which sufficient training data would otherwise be difficult to obtain. In applications of autonomous driving, this could include hazardous situations, such as nearby accidents or challenging weather or lighting conditions. Examples are particularly well-suited for modeling the overall variability in recorded training data, which could include, for example, sensor properties, weather conditions, etc. GANs can also be used to generate specific training data for applications such as object detection, where hard positive or negative cases are needed to further improve an already working detector. Positive and negative cases can be modeled as two distinct domains.

[0253] GANs can be used to enhance recorded data by adding features that are missing during recording but are part of the expected data distribution. This includes applications such as image in-painting, for example, filling in missing regions in an image (e.g., due to sensor defects), such as de-occlusion when parts of an important object are hidden by another object (e.g., a pedestrian hidden behind a vehicle) and complicate classification. It can also be used to transform hand-drawn drawings or annotated diagrams into realistic images, for example, to construct specific extreme cases. GANs can also be used to enhance the resolution of images recorded using low-resolution sensors.

[0254] The applications described above are not limited to images but can also be applied to image-like data, such as sound spectrograms, radar spectra, and ultrasound images, and therefore can be used to generate synthetic data for other domains, such as speech synthesis. Embodiments can be conditioned on other sensor inputs or abstract features, and therefore can also be used for multi-sensory fusion, for example, to convert between different data modalities / sensors.

[0255] The proposed framework can also be used for future state prediction, which is particularly important for the successful operation of autonomous agents in the real world (e.g., to anticipate the movement of pedestrians and vehicles used for autonomous driving). The ability to anticipate future scene states—which involves mapping a scene state to possible future states under uncertainty—is a key capability of autonomous agents.

[0256] Another application of the embodiments is de-occlusion. For example, in an embodiment, multiple transformation targets include information indicating an object identified in sensor data, and an indication of whether the object is to be occluded or de-occluded. Figure 6a An example of an embodiment of the measured sensor data is illustrated schematically. Figure 6aThe image shows a person behind a car (e.g., obscured by the car). Figure 6b The same scenario is illustrated, but here, the person moves from behind the car to in front of it. The person is no longer occluded by the car; for example, the person has been deoccluded. The deocclusion operation can be performed by an embodiment of the generator network. For example, images 6a and 6b can be taken as sensor data. The transformation objective includes whether to occlude or deocclude. For example, the transformation objective could be as follows: Figure 6c Like in the middle. In this case, the black bar indicates that the person should be unobstructed. For example, Figure 6a and 6c An image can be provided as input to a generator image, which can be trained to produce image 6b as a response. The generator network can be configured to perform the inverse transformation. For example, it can be used on different channels... Figure 6c A similar image is used to instruct the network to occlude a person. Given this target and image 6b, image 6a can be generated. This embodiment also provides examples of embodiments in which the number of targets being transformed can be very large.

[0257] In this scenario, the discriminator network can distinguish between real and generated images and verify that the person indicated by the black bar is indeed occluded or de-occluded.

[0258] The generator and discriminator networks may be trained on unpaired data. For example, Figure 6d An example of an embodiment of the measured sensor data is illustrated schematically, and Figure 6e An example of an embodiment of a transformation target is illustrated schematically. Figure 6d Images and Figure 6e The transformation target can be used as an example in the class of people to be de-occluded. A similar case can be used to define the class of people to be occluded. The transformation target can be generated automatically. Automatically identifying people to be occluded or de-occluded is much easier than automatically occluding or de-occluding them. For example, a standard object classifier can be used to obtain the former.

[0259] Figure 7a An example of an embodiment of a training method 700 is schematically illustrated. This training method 700 is configured to train a generator neural network arranged to transform measured sensor data into generated sensor data. Method 700 includes:

[0260] -Access the training set of sensor data measured by (710),

[0261] - Training (720) the generator network along with the discriminator neural network includes optimizing (730) the generator network to transform measured sensor data into generated sensor data, and optimizing (740) the discriminator network to distinguish between measured sensor data and generated sensor data, wherein the training includes

[0262] - Obtain sensor data (750) as input and a transformation target selected from multiple transformation targets, and transform (760) the sensor data according to the transformation target using a generator network, and

[0263] - Obtain (770) sensor data as input and a transformation target selected from the plurality of transformation targets, and use a discriminator network to determine (780) whether the sensor data is measured sensor data that satisfies the transformation target.

[0264] Figure 7b An example embodiment of a generator method 800 is schematically illustrated, which is arranged to transform measured sensor data into generated sensor data. Method 800 includes:

[0265] - Receive (810) the measured sensor data and the transformed target as input,

[0266] - The trained generator network (820) is applied to the received input measured sensor data and the transformed target, wherein, according to an embodiment, the generator network has been trained.

[0267] As will be apparent to those skilled in the art, many different ways are possible to perform this method. For example, the steps can be performed in the order shown, but the order of the steps can also be changed, or some steps can be performed in parallel. Furthermore, other method steps can be inserted between steps. The inserted steps may represent a refinement of the method as described herein, or they may be unrelated to this method. For example, most of the training can be performed at least partially and in parallel. Moreover, a given step may not be fully completed before the next step begins.

[0268] Embodiments of the present invention can be executed using software that includes instructions for causing a processor system to perform method 700 or 800. The software may include only those steps taken by a specific sub-entity of the system. The software can be stored in a suitable storage medium, such as a hard disk, floppy disk, memory, optical disk, etc. The software can be transmitted via wired or wireless means or using a data network (e.g., the Internet) as a signal. This makes the software available for download and / or for remote use on a server. Embodiments of the method can be executed using a bitstream arranged to configure programmable logic (e.g., a field-programmable gate array (FPGA)) to implement the method.

[0269] It will be understood that the invention is also extended to computer programs adapted to put the invention into practice, particularly computer programs on or within a carrier. Such programs may take the form of source code, object code, intermediate source code, and object code such as partially compiled form, or any other form suitable for use in implementations of embodiments of the method. Embodiments relating to the computer program product include computer-executable instructions corresponding to each processing step of at least one of the illustrated methods. These instructions may be subdivided into subroutines and / or stored in one or more files that may be statically or dynamically linked. Another embodiment relating to the computer program product includes computer-executable instructions that correspond to each component of at least one of the illustrated systems and / or products.

[0270] Figure 8a A computer-readable medium 1000 with a writable portion 1010 according to an embodiment is shown. The writable portion 1010 includes a computer program 1020, which includes instructions for causing a processor system to perform a training or generation method. The computer program 1020 may be embodied on the computer-readable medium 1000 as a physical label or by means of magnetization of the computer-readable medium 1000. However, any other suitable embodiments are conceivable. Furthermore, it will be appreciated that although the computer-readable medium 1000 is shown herein as an optical disc, the computer-readable medium 1000 may be any suitable computer-readable medium (such as a hard disk, solid-state storage, flash memory, etc.) and may be non-recordable or recordable. The computer program 1020 includes instructions for causing a processor system to perform the training or generation method.

[0271] Figure 8b A schematic representation of a processor system 1140 according to an embodiment of a training system and / or a generation system is shown. The processor system includes one or more integrated circuits 1110. Figure 8bThe diagram schematically illustrates the architecture of one or more integrated circuits 1110. Circuit 1110 includes a processing unit 1120 (e.g., a CPU) for running computer program components to perform methods according to embodiments and / or implement modules or units thereof. Circuit 1110 includes a memory 1122 for storing programming code, data, etc. A portion of the memory 1122 may be read-only. Circuit 1110 may include a communication element 1126, such as an antenna, a connector, or both. Circuit 1110 may include an application-specific integrated circuit 1124 for performing some or all of the processing defined in the method. Processor 1120, memory 1122, application-specific IC 1124, and communication element 1126 may be interconnected to each other via interconnect 1130 (e.g., a bus). Processor system 1110 may be arranged for contact and / or contactless communication using antennas and / or connectors, respectively.

[0272] For example, in an embodiment, processor system 1140 (e.g., an embodiment of a training system and / or a generation system) may include processor circuitry and memory circuitry, with the processor arranged to execute software stored in the memory circuitry. For example, the processor circuitry may be an Intel Core i7 processor, an ARM Cortex-R8, etc. In an embodiment, the processor circuitry may be an ARM Cortex M0. The memory circuitry may be ROM circuitry or non-volatile memory, such as flash memory.

[0273] It should be noted that the embodiments mentioned above are illustrative and not limiting of the invention, and those skilled in the art will be able to devise many alternative embodiments.

[0274] In the claims, any reference marks placed between parentheses should not be construed as limiting the claims. The use of the verb "comprising" and its variations does not exclude the presence of elements or steps other than those recited in the claims. The article "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. Expressions such as "at least one of the following"—when preceding a list of elements—indicate the selection of all elements or any subset of elements from the list. For example, the expression "at least one of A, B, and C" should be understood to include: only A; only B; only C; both A and B; both A and C; both B and C; or all A, B, and C. The invention can be implemented by means of hardware comprising several different elements, and by means of a suitably programmed computer. In a device claim enumerating several components, several of these components may be embodied by one and the same item of hardware. The fact that certain measures are recited in mutually different dependent claims does not indicate that combinations of these measures cannot be advantageously used.

[0275] In the claims, references enclosed in parentheses refer to reference numerals in the drawings of illustrative embodiments or formulas of embodiments, thus increasing the comprehensibility of the claims. These references should not be construed as limiting the claims.

Claims

1. A training system (200) for training a generator neural network (100; 220), said generator neural network (100; 220) being arranged to transform measured sensor data into generated sensor data (223), said system comprising - A communication interface for accessing the training set (351; 352; 353) of the measured sensor data, where the measured sensor data is image or video data; - A processor system, configured to train a generator network along with a discriminator neural network (230; 238; 239), wherein the generator network is optimized to transform measured sensor data into generated sensor data, and the discriminator network is optimized to distinguish between the measured sensor data and the generated sensor data. - The generator network is arranged to receive sensor data as input and a transformation target selected from a plurality of transformation targets, and is arranged to transform the sensor data according to the transformation target; - The discriminator network is arranged to receive two sensor data (234; 235), each of which is measured sensor data or generated sensor data, as input, and a transformation target selected from the plurality of transformation targets, and to determine whether the sensor data (234; 235) are both measured sensor data, and whether the change from sensor data (234) to sensor data (235) corresponds to the transformation indicated by the transformation target.

2. The system according to claim 1, wherein The multiple transformation targets include multiple domains, the training set is labeled using domains from the multiple domains, the generator network is configured to transform sensor data to a domain according to the transformation targets, and the discriminator network is configured to determine whether the input sensor data satisfies the domain according to the transformation targets, or The multiple transformation targets include multiple time differences, the training set is labeled using timestamps, the generator network is configured to transform sensor data from a first timestamp to a second timestamp according to the time difference, and the discriminator network is configured to receive first sensor data, second sensor data and time differences as input, and determine whether the first sensor data and the second sensor data satisfy the time difference.

3. The system according to claim 1 or 2, wherein, The discriminator network is trained to determine the sensor data and the transformation target based on the training set.

4. The system according to claim 1 or 2, wherein, The discriminator network is trained to determine whether the input sensor data is the generated sensor data and / or whether the input sensor data is not based on the input transformation target.

5. The system according to claim 1 or 2, wherein, The training set includes one or more sets of first measured sensor data, transformation target, and second measured sensor data. The generator network is trained to transform the first measured sensor data into the second measured sensor data according to the transformation target.

6. The system according to claim 1 or 2, wherein, The training set includes images from time-lapse videos labeled with corresponding timestamps.

7. The system according to claim 1 or 2, wherein, The plurality of transformation targets include: information indicating objects identified in the sensor data, and an indication of whether the objects are to be occluded or unoccluded.

8. The system according to claim 1 or 2, wherein, - The generator network and / or discriminator network includes one or more neurons that receive at least a portion of the sensor data and at least a portion of the transformed target, and / or - The generator network and / or discriminator network are arranged to receive multiple channels as input, at least one of which encodes the sensor data and at least one of the channels encodes the transformed target, and / or - The generator network and / or discriminator network include multiple layers that receive the transformation target, and / or - The generator network and / or discriminator network include an encoding block, a processing block, and / or a decoding block, at least two of which receive the transformation target as input.

9. The system according to claim 1 or 2, wherein, The plurality of transformation targets includes at least one multi-transformation target, such that the measured sensor data is transformed sequentially according to the multi-transformation target.

10. A training system (200) for training multiple individual generator neural networks (100; 220), each generator neural network being arranged to transform measured sensor data into generated sensor data (223), said system comprising - Communication interface for accessing the training set (351; 352; 353) of the measured sensor data, where the measured sensor data is image or video data. - A processor system configured to train multiple generator networks along with discriminator neural networks (230; 238; 239), wherein the generator networks are optimized to transform measured sensor data into generated sensor data, and the discriminator networks are optimized to distinguish between the measured sensor data and the generated sensor data. - Each of the plurality of generator networks is associated with a corresponding transformation target and is arranged to receive sensor data as input and transform the sensor data according to the transformation target, wherein - The discriminator network is arranged to receive two sensor data (234; 235), each of which is measured sensor data or generated sensor data, as input, and a transformation target selected from a plurality of transformation targets, and to determine whether the sensor data (234; 235) are both measured sensor data, and whether the change from sensor data (234) to sensor data (235) corresponds to the transformation indicated by the transformation target.

11. A generator system for a generator neural network, the generator neural network being arranged to transform measured sensor data into generated sensor data, the system comprising: - Communication interface, used to receive measured sensor data and transformed target as input. - A processor system configured to apply a trained generator network to the received input of measured sensor data and the transformed target, wherein, The generator network has been trained using the system according to any one of claims 1-9.

12. The generator system according to claim 11, wherein, The sensor data includes vehicle sensor data, and wherein The transformation objective includes a time difference to generate sensor data based on different times of day and / or year, and / or - The transformation target includes a domain that indicates the vehicle's environmental type, and / or - The transformation target includes the type of sensor data. The target of the transformation includes the color or model of the car.

13. A training method (700) for training a generator neural network, said generator neural network being arranged to transform measured sensor data into generated sensor data, said method comprising: - Access the training set of the measured sensor data (710), where the measured sensor data is image or video data. - Training (720) the generator network along with the discriminator neural network includes optimizing (730) the generator network to transform measured sensor data into generated sensor data, and optimizing (740) the discriminator network to distinguish between measured sensor data and generated sensor data, wherein the training includes - Obtain (750) sensor data as input and a transformation target selected from multiple transformation targets, and transform (760) the sensor data according to the transformation target using a generator network; - Obtain (770) two sensor data (234; 235), each of which is measured sensor data or generated sensor data, as input, and a transformation target selected from the plurality of transformation targets, and use a discriminator network to determine (780) whether the sensor data (234; 235) are both measured sensor data, and whether the change from sensor data (234) to sensor data (235) corresponds to the transformation indicated by the transformation target.

14. A temporary or non-temporary computer-readable medium (1000) comprising data (1020) representing instructions for causing a processor system to perform the method according to claim 13.

15. A temporary or non-temporary computer-readable medium (1000) comprising data (1020) representing a generator neural network and a discriminator neural network according to any one of claims 1-13.

Images