Method for generating a training dataset

Abstract

A computer-implemented method for generating a training dataset includes generating a plurality of synthetic image datasets based on a corresponding plurality of input image datasets and a target factor, using a generator model. Each synthetic image dataset of the plurality of synthetic image datasets is at least substantially associated with a target factor. The method further includes combining the plurality of synthetic image datasets to result in a unified training dataset associated with the target factor.

Description

METHOD FOR GENERATING A TRAINING DATASETTECHNICAL FIELD

[0001] Various embodiments relate to methods for generating a training dataset.BACKGROUND

[0002] The training of camera-based neural networks requires large amounts of annotated data, such as annotated images. However, annotating of the data, such as 3D data, is expensive and time consuming. While there may be various small, annotated datasets made available by different sources, these datasets are captured using different sensors, and hence exhibit different sensor characteristics. These datasets cannot be combined to form a larger dataset for training the neural network. In view of the above, there is a need for an improved method of generating a training dataset that can address at least some of the abovementioned problems.SUMMARY

[0003] According to various embodiments, a computer-implemented method for generating a training dataset may be provided. The method includes generating a plurality of synthetic image datasets based on a corresponding plurality of input image datasets and a target factor, using a generator model. Each synthetic image dataset may be at least substantially associated with the target factor. The method also includes combining the plurality of synthetic image datasets to result in a unified training dataset associated with the target factor.

[0004] According to various embodiments, a computer program product may be provided. The computer program product may include instructions, which, when the computer program product is executed by a computer, cause the computer to carry out the abovementioned method for generating a training dataset.

[0005] According to various embodiments, a data carrier signal may be provided. The data carrier signal may carry the abovementioned computer program product.

[0006] Additional features for advantageous embodiments are provided in the dependent claims.BRIEF DESCRIPTION OF THE DRAWINGS

[0007] In the drawings, like reference characters generally refer to the same parts throughout the different views. The drawings are not necessarily to scale, emphasis instead generally being placed upon illustrating the principles of the invention. In the following description, various embodiments are described with reference to the following drawings, in which:

[0008] FIG. l is a simplified block diagram that illustrates a method for generating a training dataset according to various embodiments.

[0009] FIG. 2 shows a flow diagram of the method for generating a training dataset according to various embodiments.

[0010] FIG. 3 shows a block diagram of a computer program product for generating a training dataset according to various embodiments.

[0011] FIG. 4 shows a simplified block diagram that illustrates a method for generating a training dataset according to an embodiment.DESCRIPTION

[0012] Embodiments described below in context of the devices are analogously valid for the respective methods, and vice versa. Furthermore, it will be understood that the embodiments described below may be combined, for example, a part of one embodiment may be combined with a part of another embodiment.

[0013] It will be understood that any property described herein for a specific device may also hold for any device described herein. It will be understood that any property described herein for a specific method may also hold for any method described herein. Furthermore, it will be understood that for any device or method described herein, not necessarily all the components or steps described must be enclosed in the device or method, but only some (but not all) components or steps may be enclosed.

[0014] The term “coupled” (or “connected”) herein may be understood as electrically coupled or as mechanically coupled, for example attached or fixed, or just in contact without any fixation, and it will be understood that both direct coupling or indirect coupling (in other words: coupling without direct contact) may be provided.

[0015] In this context, the device as described in this description may include a memory which is for example used in the processing carried out in the device. A memory used in theembodiments may be a volatile memory, for example a DRAM (Dynamic Random Access Memory) or a non-volatile memory, for example a PROM (Programmable Read Only Memory), an EPROM (Erasable PROM), EEPROM (Electrically Erasable PROM), or a flash memory, e.g., a floating gate memory, a charge trapping memory, an MRAM (Magnetoresistive Random Access Memory) or a PCRAM (Phase Change Random Access Memory).

[0016] In order that the invention may be readily understood and put into practical effect, various embodiments will now be described by way of examples and not limitations, and with reference to the figures.

[0017] According to various embodiments, a method 100 for generating a training dataset is provided. The method 100 may include converting a plurality of input images to adopt a predefined “style”, so that these images may be combined into a single dataset that uniformly exhibits the predefined “style”. The “style” is referred herein interchangeably as “factor”. The “predefined ‘style’ ” is referred herein interchangeably as the “target factor”. The conversion of these input images may be performed by a Generative Adversarial Networks (GAN). The target factor may include sensor parameters such as camera intrinsic or extrinsic. The target factor may include environmental parameters such as lighting condition or weather condition. The resulting dataset may be used as a training dataset, to train machine learning models.

[0018] FIG. l is a simplified block diagram that illustrates the method 100 according to various embodiments. The method 100 may include inputting a plurality of input image datasets 110 to a generator model 120 for generating a corresponding plurality of synthetic image datasets 130. The plurality of input image datasets 110 may include two or more input image datasets, for example, a first input image dataset 112, a second input image dataset 114 and a third input image dataset 116. These input image datasets 110 may be associated with different factors. For example, the first input image dataset 112 may be associated with, or in other words, exhibits, a first factor j , the second input image dataset 114 may be associated with a second factor f2and the third input image dataset 116 may be associated with a third factor f3. The generator model 120 may include a GAN. The generator model 120 may further receive a target factor data. The target factor data may include an image dataset associated with a target factor 190, or may include the target factor 190, depending on the type of GAN present in the generator model 120.

[0019] The generator model 120 may output a first synthetic image dataset 132 based on the first input image dataset 112 and further based on the target factor 190. The generator model120 may output a second synthetic image dataset 134 based on the second input image dataset 114 and further based on the target factor 190. The generator model 120 may output a third synthetic image dataset 136 based on the third input image dataset 116 and further based on the target factor 190. The first synthetic image dataset 132, the second synthetic image dataset 134 and the third synthetic image dataset 136 may be combined to form a unified training dataset 150 by a combiner 140 if the synthetic images passed the examination of the discriminator model 160. These synthetic image datasets 130 may at least substantially exhibit the target factor 190. The combiner 140 may combine the synthetic image datasets 130 by collating them or by concatenating them if they passed the quality check of the discriminator model 160.

[0020] The generator model 120 may be configured to generate the synthetic image datasets 130 by transforming the input image dataset 110 based on the target factorl90. The generator model 120 may supervise the target factor 190 exhibited in the synthetic image datasets 130. For example, the target factor 190 may include a target camera pose, i.e., camera extrinsic parameters. The target factor 190 may also include target camera intrinsic parameters. The generator model 120 may receive input images captured at other camera poses and may geometrically transform the input images to take on the perspective of the target camera pose. Additionally, or alternatively, the input images may also be captured using a first set of camera intrinsic parameters and transformed to a second set of camera intrinsic parameters.

[0021] An example of such a generator model 120 may be a conditional generative adversarial net (cGAN), a CycleGAN, or combinations thereof, concept. An implementation of cGAN is disclosed in “Conditional Generative Adversarial Nets” by Mirza et. Al. An implementation of CycleGAN is disclosed in “Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks” by Zhu et. Al., which can be accessed at https: / / junyanz.github.io / CycleGAN / .

[0022] In an embodiment, the generator model 120 may include an encoder-decoder structure. The encoder may create a representation of the input image. The decoder may receive the representation of the input image and may generate the synthetic image based on the representation, the target factor 190 and optionally noise.

[0023] Cameras capture images, and the captured images are visual projections of a 3D environment onto a 2D plane. The transformation of the 3D data points onto the 2D plane depends on both camera intrinsic and extrinsic parameters. In an embodiment, the target factor 190 may include camera intrinsic. The camera intrinsic relate to how the camera capturesimages. The camera intrinsic may include, for example, focal length (denoted herein as ), field of view (FOV), aperture, resolution, and center offset (denoted herein as c). In another embodiment, the target factor 190 may include may refer to camera extrinsic such as the camera position and the camera orientation.

[0024] In another embodiment, the target factor 190 may include environmental conditions that the sensor is operating in, for example, weather and / or lighting conditions. For example, the generator 120 may convert an input image dataset captured in daylight to look like they were captured at night. For example, the generator 120 may convert an input image dataset captured in rainy or snowy conditions to look like they were captured on a sunny day. As a result, the generator 120 may generate synthetic datasets without requiring manual effort in setting up the sensors in a different, or even possibly difficult environment.

[0025] The amount of data in the training dataset 150 may be a sum of the amount of data in the plurality of input image datasets 110. In other words, the method 100 may result in a large training dataset that exhibits the target factor 190. The performance of a neural network depends on the quality and quantity of its training data, and as such the large training dataset 150 may provide the advantage of improving the training of a neural network such as a 3D model that uses camera images as input. The large training dataset 150 may be useful for training neural networks that process imagery, for example, for object detection, 3D free space detection, 3D lane marking detection and more. The method 100 may also be useful for generating a training dataset for which there is no available annotated dataset.

[0026] The discriminator model 160 may include a classifier, which learns to model the probability of a given input image being real or fake with the help of binary classification. The discriminator model 160 may compare the synthetic images 130 to the real images 170 associated with the target factor 190, to determine a probability that the synthetic images 130 are real, i.e. not generated by the generator model 120. The discriminator model 160 may serve to reject synthetic images that do not match with the target factor 190.

[0027] According to various embodiments, the discriminator model 160 may be configured to penalize structure at the scale of local image patches and may attempt to classify each image patch as either real or fake. The discriminator model 160 may be run convolutionally across the image, to average all responses to provide the output. An example of the discriminator model 160 may be the PatchGAN as disclosed in “Image-to-Image Translation with Conditional Adversarial Networks” by Isola et. al., which is incorporated herein by reference.

[0028] According to various embodiments, the generator model 120 and the discriminator model 160 may be trained together. The training process may include inputting training data (i.e., real images 170) to the discriminator model 160. The discriminator model 160 may be trained to maximize the probability of assigning the correct label to both the real images 170 and the synthetic images 130 generated by the generator model 120. Simultaneously, the generator model 120 may be trained to minimize the probability of its output (i.e., the synthetic images 130) being classified as fake. The discriminator model 160 may determine a probability that indicates how close the factor of the synthetic images 130 are, as compared to those of the real images 170. The output probability from the discriminator model 160 may serve as the loss function for the generator model 120. The generator model 120 may adapt its weights according to the loss function, to improve the accuracy of generating synthetic images with the target factor. This adversarial training strategy may improve the accuracy of generating synthetic image datasets 130 that matches the target sensor parameters or a unified set of sensor parameters amongst the synthetic image datasets 130.

[0029] The training process may be repeated until the discriminator model 160 is unable to differentiate between the real images 170 and the synthetic images 130. An example of the training process is disclosed in “Generative Adversarial Nets” by Goodfellow et. al., accessible at https: / / arxiv.org / pdf / 1406.2661.pdf, which is incorporated herein by reference. An example of a suitable source for the real images 170 may be the nuScenes dataset.

[0030] FIG. 2 shows a flow diagram of the method 100 for generating a training dataset according to various embodiments. The method 100 may include processes 202 and 204. The process 202 may include generating a plurality of synthetic image datasets 130 based on a corresponding plurality of input image datasets 110 and a target factor 190, using the generator model 120. Each synthetic image dataset of the plurality of synthetic image datasets 130 may be associated with the target factor 190. The process 202 may include combining the plurality of synthetic image datasets 130 to result in a unified training dataset 150 that is associated with the target factor 190. The method 100 may thereby unify the factors of a plurality of image datasets such that these image datasets can then be combined into a larger image dataset to be used for training neural networks. The accuracy and performance of the neural networks improve with the increased amount of information that is used for the training. The method 100 may also be used to generate a training dataset associated with a factor that does not have readily available annotated dataset.

[0031] According to an embodiment which may be combined with any above-described embodiment or with any below described further embodiment, the method 100 may further include receiving a target image dataset associated with the target factor 190, in the generator model 120. The method 100 may further include generating the plurality of synthetic image datasets by transforming the plurality of input image datasets based on the target image dataset, using the generator model 120. The generator model 120 may thus be able to generate synthetic image datasets without having details of the target factor 190, which may be unknown in some situations.

[0032] According to an embodiment which may be combined with any above-described embodiment or with any below described further embodiment, the method 100 may further include receiving the target factor 190 in the generator model. The method 100 may further include generating the plurality of synthetic image datasets by transforming the plurality of input image datasets based on the received target factor 190, using the generator model 120. The generator model 120 may thus be able to generate synthetic image datasets that match the target factor 190 accurately.

[0033] According to an embodiment which may be combined with any above-described embodiment or with any below described further embodiment, the method 100 may further include comparing the plurality of synthetic image datasets 130 against the real images 170 associated with the target factor, using a discriminator model 160, and combining the plurality of synthetic image datasets to result in the unified training dataset 150 based on the comparison. The discriminator model 160 may identify synthetic images that do not sufficiently exhibit the target factor, and as such, the uniformity of the unified training dataset 150 in being associated with the target factor, may be enhanced.

[0034] According to an embodiment which may be combined with any above-described embodiment or with any below described further embodiment, the real images 170 may be unpaired with the plurality of input image datasets 110. The generator model 120 can be trained even without any ground truth data that matches the input image datasets 110 to the target factor 190.

[0035] According to an embodiment which may be combined with any above-described embodiment or with any below described further embodiment, the generator model 120 may include a GAN.

[0036] According to an embodiment which may be combined with any above-described embodiment or with any below described further embodiment, the GAN may include one of CycleGAN, Conditional GAN and PatchGAN.

[0037] According to an embodiment which may be combined with any above-described embodiment or with any below described further embodiment, the target factor 190 may include camera intrinsic parameters. This allows image datasets generated using different sensor types, to be combined.

[0038] According to an embodiment which may be combined with any above-described embodiment or with any below described further embodiment, the target factor 190 may include camera extrinsic parameters. This allows image datasets that are generated by differently positioned sensors, to be combined.

[0039] According to an embodiment which may be combined with any above-described embodiment or with any below described further embodiment, the target factor 190 may include at least one of lighting condition and weather condition.

[0040] According to an embodiment which may be combined with any above-described embodiment or with any below described further embodiment, each synthetic image dataset 130 may be a geometric transformation of the corresponding input image dataset 110.

[0041] FIG. 3 shows a block diagram of a computer program product 300 for generating a training dataset according to various embodiments. The computer program product 300 may include the generator model 120, the discriminator model 160 and the combiner 140. The computer program product may be executable by a computer, to carry out the method 100.

[0042] According to various embodiments, a data carrier signal may be provided. The data carrier signal may include the computer program product 300.

[0043] FIG. 4 shows a simplified block diagram that illustrates the method 100 according to another embodiment. The generator model 120 may include a generator model A 420 and a generator model B 422. The method 100 may include inputting input image dataset 110 to the generator model A 420. The generator model A 420 may further receive a target factor data. The target factor data may include the target factor 190, or conditional input. The generator model 120 may output a synthetic image dataset 130 that may at least substantially exhibit the target factor 190.

[0044] The method 100 may further include inputting the synthetic image dataset 130 to the generator model B 422. The generator model B 422 may recreate the input dataset based on the synthetic image dataset 130. The recreated input dataset 402 generated by the generatormodel B 422 may at least substantially exhibit the input image factor 192 and may be provided to a discriminator model B 462. The discriminator model B 462 may further receive the real image dataset 110 associated with the real images factor 110 or conditional input, to compare the recreated input dataset 402 against the real images 110. The discriminator model B 462 may determine a probability that indicates how close the factor of the recreated input dataset 402 is, as compared to those of the real images 110. The output probability from the discriminator model B 462 may serve as the loss function for the generator model B 422. The generator model B 422 may adapt its weights according to the loss function, to improve the accuracy of generating synthetic images with the target factor.

[0045] The method 100 may further include providing synthetic image 130 to a discriminator model A 460. The discriminator model A 460 may further receive the real images 170 associated with the target factor 190 and the target factor 190 or conditional input, to compare the synthetic image 130 against the real images 170. The discriminator model A 460 may determine a probability that indicates how close the factor of the synthetic image is, as compared to those of the real images 170. The output probability from the discriminator model A 460 may serve as the loss function for the generator model A 420. The generator model A 420 may adapt its weights according to the loss function, to improve the accuracy of generating synthetic images with the target factor. After the model training stage and during the inference stage, if the discriminator model A 460 cannot classify the synthetic image 130 is synthetic, then the image 130 will be added to the training dataset 150.

[0046] In an embodiment, the generator model A 420 and the discriminator model A 460 may be part of one cGAN, while the generator model B 422 and discriminator model B 462 may be part of another cGAN. The generator model A 420, B 422 and the discriminator model B460 and B462 may be part of a CycleGAN.

[0047] While embodiments of the invention have been particularly shown and described with reference to specific embodiments, it should be understood by those skilled in the art that various changes in form and detail may be made therein without departing from the spirit and scope of the invention as defined by the appended claims. The scope of the invention is thus indicated by the appended claims and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced. It will be appreciated that common numerals, used in the relevant drawings, refer to components that serve a similar or the same purpose.

[0048] It will be appreciated to a person skilled in the art that the terminology used herein is for the purpose of describing various embodiments only and is not intended to be limiting of the present invention. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0049] It is understood that the specific order or hierarchy of blocks in the processes / flowcharts disclosed is an illustration of exemplary approaches. Based upon design preferences, it is understood that the specific order or hierarchy of blocks in the processes / flowcharts may be rearranged. Further, some blocks may be combined or omitted. The accompanying method claims present elements of the various blocks in a sample order and are not meant to be limited to the specific order or hierarchy presented.

[0050] The previous description is provided to enable any person skilled in the art to practice the various aspects described herein. Various modifications to these aspects will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other aspects. Thus, the claims are not intended to be limited to the aspects shown herein but is to be accorded the full scope consistent with the language claims, wherein reference to an element in the singular is not intended to mean “one and only one” unless specifically so stated, but rather “one or more.” The word “exemplary” is used herein to mean “serving as an example, instance, or illustration.” Any aspect described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other aspects. Unless specifically stated otherwise, the term “some” refers to one or more. Combinations such as “at least one of A, B, or C,” “one or more of A, B, or C,” “at least one of A, B, and C,” “one or more of A, B, and C,” and “A, B, C, or any combination thereof’ include any combination of A, B, and / or C, and may include multiples of A, multiples of B, or multiples of C. Specifically, combinations such as “at least one of A, B, or C,” “one or more of A, B, or C,” “at least one of A, B, and C,” “one or more of A, B, and C,” and “A, B, C, or any combination thereof’ may be A only, B only, C only, A and B, A and C, B and C, or A and B and C, where any such combinations may contain one or more member or members of A, B, or C. All structural and functional equivalents to the elements of the various aspects described throughout this disclosure that are known or latercome to be known to those of ordinary skill in the art are expressly incorporated herein by reference and are intended to be encompassed by the claims.

Claims

CLAIMS1. A computer-implemented method (100) for generating a training dataset (150), the method (100) comprising: generating a plurality of synthetic image datasets (130) based on a corresponding plurality of input image datasets (110) and a target factor, using a generator model (120), wherein each synthetic image dataset (130) of the plurality of synthetic image datasets (130) is at least substantially associated with a target factor (190); and combining the plurality of synthetic image datasets (130) to result in a unified training dataset (150) associated with the target factor (190).

2. The method (100) of any preceding claim, further comprising: receiving a target image dataset associated with the target factor (190), in the generator model (120); and generating the plurality of synthetic image datasets (130) by transforming the plurality of input image datasets (110) based on the target image dataset, using the generator model (120).

3. The method (100) of any preceding claim, further comprising: receiving the target factor (190) in the generator model (120); and generating the plurality of synthetic image datasets (130) by transforming the plurality of input image datasets (110) based on the received target factor (190), using the generator model (120).

4. The method (100) of any preceding claim, further comprising: comparing the plurality of synthetic image datasets (130) against real images (170) associated with the target factor (190), using a discriminator model (160); and combining the plurality of synthetic image datasets (130) to result in the unified training dataset (150) based on the comparison.

5. The method (100) of claim 4, wherein the real images (170) are unpaired with the plurality of input image datasets (110).

6. The method (100) of any preceding claim, wherein the generator model (120) comprises a generative adversarial network.

7. The method (100) of claim 5, wherein the generative adversarial network comprises one of CycleGAN, Conditional GAN and PatchGAN.

8. The method (100) of any preceding claim, wherein the target factor (190) comprises camera intrinsic parameters.

9. The method (100) of any preceding claim, wherein the target factor (190) comprises camera extrinsic parameters.

10. The method (100) of any preceding claim, wherein the target factor (190) comprises at least one of lighting condition and weather condition.

11. The method (100) of any preceding claim, wherein each synthetic image dataset (130) is a geometric transformation of the corresponding input image dataset (110).

12. A computer program product (300) comprising instructions, which, when the computer program product (300) is executed by a computer, cause the computer to carry out the method (100) of any preceding claims.

13. A data carrier signal carrying the computer program product (300) of claim 12.

Images