Semantically consistent augmented training data for traffic light detection

Abstract

Semantically consistent augmented training data for traffic light detection. Methods, systems, and non-transitory computer-readable media for generating augmented data to train a deep neural network to detect traffic lights in image data. The method includes receiving a plurality of real-world road scene images, and selecting a subset of the plurality of real-world road scene images. The method also includes selecting an image from the subset, and determining a distribution indicating a likelihood that each location in the selected image can contain a traffic light. The method further includes selecting a location in the selected image by sampling the distribution, and superimposing a traffic light image onto the selected image at the selected location to generate an augmented road scene image. The method also includes processing each image in the subset to generate a plurality of augmented road scene images. The method further includes training a deep neural network model using the plurality of real-world and augmented road scene images.

Description

TECHNICAL FIELD

[0001] The present disclosure relates generally to data augmentation for training object detection models. More specifically, the present disclosure relates to data augmentation for training traffic light detection models in autonomous driving applications. BACKGROUND

[0002] Road scene understanding is important in autonomous driving applications. Traffic light detection plays an important role in road scene understanding. However, detecting traffic lights in image data is challenging because traffic lights are usually small in size, located far away from the detector, and have very low contrast compared to their surrounding environment. SUMMARY

[0003] Deep neural networks are used to detect traffic lights in image data. Deep neural networks require a large amount of labeled training data to avoid overfitting. However, it is often difficult to obtain a large amount of labeled training data. Data augmentation is used to overcome the scarcity of labeled training data. Accordingly, the present disclosure provides, among other things, methods, systems, and non-transitory computer-readable media for generating augmented data to train a deep neural network to detect traffic lights in image data.

[0004] The present disclosure provides a method for generating augmented data to train a deep neural network to detect traffic lights in image data. The method includes a step (a) of receiving a plurality of real-world road scene images. The method also includes a step (b) of selecting a subset of real-world road scene images from the plurality of real-world road scene images. The method further includes a step (c) of selecting a real-world road scene image from the subset of real-world road scene images. The method also includes a step (d) of determining a distribution indicating a likelihood that each location in the selected real-world road scene image can contain a traffic light. The method further includes a step (e) of selecting a location in the selected real-world road scene image by sampling the distribution. The method further includes a step (f) of superimposing a traffic light image onto the selected real-world road scene image at the selected location to generate an augmented road scene image. The method also includes a step (g) of repeating steps (b) through (f) for each real-world road scene image in the subset of real-world road scene images to generate a plurality of augmented road scene images. The method further includes a step (h) of training a pre-trained deep neural network model using the plurality of real-world road scene images and the plurality of augmented road scene images to generate a newly trained deep neural network model.

[0005] The present disclosure also provides a system for generating augmented data to train a deep neural network to detect traffic lights in image data. In one embodiment, the system includes an electronic processor and a memory. The memory stores computer-readable instructions that, when executed by the electronic processor, cause the electronic processor to perform operations. The operations include an operation (a) of receiving a plurality of real-world road scene images. The operations also include an operation (b) of selecting a subset of real-world road scene images from the plurality of real-world road scene images. The operations further include an operation (c) of selecting a real-world road scene image from the subset of real-world road scene images. The operations also include an operation (d) of determining a distribution indicating a likelihood that each location in the selected real-world road scene image can contain a traffic light. The operations further include an operation (e) of selecting a location in the selected real-world road scene image by sampling the distribution. The operations also further include an operation (f) of superimposing a traffic light image onto the selected real-world road scene image at the selected location to generate an augmented road scene image. The operations also further include an operation (g) of repeating operations (b) through (f) for each real-world road scene image in the subset of real-world road scene images to generate a plurality of augmented road scene images. The operations also include an operation (h) of training a pre-trained deep neural network model using the plurality of real-world road scene images and the plurality of augmented road scene images to generate a newly trained deep neural network model.

[0006] The present disclosure also provides a non-transitory computer-readable medium storing computer-readable instructions that, when executed by an electronic processor of a computer, cause the computer to perform operations. The operations include an operation (a) of receiving a plurality of real-world road scene images. The operations also include an operation (b) of selecting a subset of real-world road scene images from the plurality of real-world road scene images. The operations further include an operation (c) of selecting a real-world road scene image from the subset of real-world road scene images. The operations also include an operation (d) of determining a distribution indicating a likelihood that each location in the selected real-world road scene image can contain a traffic light. The operations further include an operation (e) of selecting a location in the selected real-world road scene image by sampling the distribution. The operations also include an operation (f) of superimposing a traffic light image onto the selected real-world road scene image at the selected location to generate an augmented road scene image. The operations also further include an operation (g) of repeating operations (b) through (f) for each real-world road scene image in the subset of real-world road scene images to generate a plurality of augmented road scene images. The operations also include an operation (h) of training a pre-trained deep neural network model using the plurality of real-world road scene images and the plurality of augmented road scene images to generate a newly trained deep neural network model. BRIEF DESCRIPTION OF DRAWINGS

[0007] The accompanying drawings, together with the detailed description below, are incorporated in and form part of the specification, and are included to further illustrate and explain the concepts and principles of the embodiments, and are a part of the detailed description. In the drawings, like reference numerals refer to like elements throughout the several views.

[0008] Figure 1 is a block diagram of one example of a system for generating augmented data to train a deep neural network to detect traffic lights in image data, in accordance with some embodiments.

[0009] Figure 2 is a block diagram of one example of a generative adversarial network (GAN) model for generating augmented training data, in accordance with some embodiments.

[0010] Figure 3A and Figure 3B is a flowchart of one example of a method for generating augmented training data using a GAN, in accordance with some embodiments.

[0011] Figure 4A and Figure 4B is a flowchart of one example of a method for generating augmented training data using prior knowledge, in accordance with some embodiments.

[0012] Figure 5 is a flowchart of one example of a method for generating augmented data to train a deep neural network to detect traffic lights in image data, in accordance with some embodiments.

[0013] Figure 6 are examples of augmented road scene images generated using different data augmentation techniques, in accordance with some embodiments.

[0014] Figure 7 is a chart of example average precision values for different data augmentation techniques, in accordance with some embodiments.

[0015] In the drawings, the system and method components have been represented where appropriate by conventional symbols for reasons of brevity and clarity, and only those specific details that are pertinent to the understanding of the embodiments have been included, in order to avoid obscuring the disclosure with details that will be readily apparent to those of ordinary skill in the art having benefit of the description herein. DETAILED DESCRIPTION

[0016] Figure 1 is a block diagram of one example of a system 100 for generating augmented data to train a deep neural network to detect traffic lights in image data. Figure 1 The system 100 illustrated in FIG. is comprised of a computer 102. Figure 1The computer 102 illustrated in the middle includes an electronic processor 104 (e.g., one or more microprocessors, an application specific integrated circuit (ASIC), a system on a chip (SoC), or other electronic controllers), a memory 106, an input / output interface 108, a user interface 110, and a bus 112. In some implementations, the computer 102 includes fewer or additional components in different configurations than those illustrated in the middle. For example, in practice, the computer 102 can include additional components such as one or more power supplies, one or more sensors, and the like. For ease of explanation, these additional components are not illustrated here. Figure 1 The computer 102 illustrated in the middle includes an electronic processor 104 (e.g., one or more microprocessors, an application specific integrated circuit (ASIC), a system on a chip (SoC), or other electronic controllers), a memory 106, an input / output interface 108, a user interface 110, and a bus 112. In some implementations, the computer 102 includes fewer or additional components in different configurations than those illustrated in the middle. For example, in practice, the computer 102 can include additional components such as one or more power supplies, one or more sensors, and the like. For ease of explanation, these additional components are not illustrated here.

[0017] The input / output interface 108 includes routines for transferring information between components within the computer 102 and components external to the computer 102. The input / output interface 108 is configured to transmit and receive data via one or more wired couplings (e.g., wires, fiber optic cables, etc.), wirelessly, or a combination thereof.

[0018] The user interface 110 includes, for example, one or more input mechanisms (e.g., a touch screen, a keypad, buttons, knobs, etc.), one or more output mechanisms (e.g., a display, a printer, a speaker, etc.), or a combination thereof. In some implementations, the user interface 110 includes a touch-sensitive interface (e.g., a touch screen display) that displays visual output generated by software applications executed by the electronic processor 104. The visual output includes, for example, graphical indicators, lights, colors, text, images, graphical user interfaces (GUIs), combinations of the foregoing, and the like. The touch-sensitive interface also receives user input using detected physical contact (e.g., detected capacitance or resistance). In some implementations, the user interface 110 is separate from the computer 102.

[0019] The bus 112 connects various components of the computer 102, including the memory 106, to the electronic processor 104. The memory 106 includes, for example, read-only memory (ROM), random access memory (RAM), electrically erasable programmable read-only memory (EEPROM), other non-transitory computer-readable media, or a combination thereof. In some implementations, the memory 106 is included in the electronic processor 104. The electronic processor 104 is configured to retrieve computer-readable instructions and data from the memory 106 and execute the computer-readable instructions to perform the functionality described herein. Figure 1 The memory 106 illustrated in the middle includes a network trainer 114, a deep neural network model 116, and image data 118.

[0020] The network trainer 114 includes computer-readable instructions that cause the electronic processor 104 to perform, among other things, the methods described herein. For example, the network trainer 114 includes computer-readable instructions that cause the electronic processor 104 to generate augmented training data and train the deep neural network model 116 to detect traffic lights in image data 118 (and other image data) stored in the memory 106.

[0021] The image data 118 includes, for example, real road scene images. The real road scene images are images taken from the perspective of a vehicle driving along a road. For example, the real road scene images can include images captured by a camera coupled to a vehicle as the vehicle drives along a road. In some implementations, each real road scene image includes at least one traffic light. In other implementations, some real road scene images do not include traffic lights.

[0022] As described above, data augmentation is used to overcome the scarcity of labeled training data. For example, traffic light images are overlaid onto real road scene images to generate augmented road scene images. The level of realism of the augmented road scene images used to train the deep neural network impacts the detection accuracy of the deep neural network. Semantic consistency increases the level of realism of the augmented training images. In particular, semantic consistency increases the level of realism of augmented training images of structured scenes, such as road scene images.

[0023] In some implementations, the augmented road scene images are generated based on the semantics of the road scene. For example, a generative adversarial network (GAN) model is used to determine a distribution of the likelihood that a location can contain a traffic light based only on the semantics of the road scene. A GAN is a class of machine learning systems in which two neural networks compete against each other in a game. For example, one neural network generates augmented road scene images that appear realistic and a second neural network evaluates the realism of the augmented road scene images. The GAN model is trained to understand the semantics of the road scene and determine the likelihood that any location can contain a traffic light. To understand the semantics of the road scene, in some implementations, the GAN model is trained using inpainted images in which a traffic light is removed from a real road scene image and the GAN model determines the presence of the removed traffic light using the original ground truth of the original image.

[0024] Figure 2 is a block diagram of one example of a GAN model 200 used to generate augmented training data. Figure 2 The GAN model 200 illustrated in FIG. 2 includes an image inpainter 202, a generator 204, a first discriminator 206, and a second discriminator 208. In some implementations, the GAN model 200 includes a first neural network and a second neural network. The first neural network includes the image inpainter 202 and the generator 204. The second neural network includes the first discriminator 206 and the second discriminator 208. Figure 2Fewer or additional components of different configuration aspects are illustrated in FIG. 1. For example, the GAN model 200 can include more than one generator and more or less than two discriminators. Moreover, in practice, the GAN model 200 can include additional components such as one or more feedback networks, one or more backpropagation networks, one or more encoders, one or more decoders, etc. These additional components are not illustrated here for ease of explanation.

[0025] The image inpainter 202 is configured to receive a real road scene image and remove one or more traffic lights included therein to generate an inpainted image with the one or more traffic lights removed. In some implementations, the image inpainter 202 sets each pixel in the real road scene image containing a traffic light to a predetermined value (e.g., zero). The inpainted image is input into the generator 204. Figure 2 The generator 204 illustrated in FIG. 1 includes a compression network 210 and a reconstruction network 212. The compression network 210 is configured to compress the content of the inpainted image into a smaller amount of content. The smaller amount of content output by the compression network 210 is the most meaningful content in the inpainted image and represents an understanding of the key connections between the locations of the traffic lights and their surrounding environment. The reconstruction network 212 is configured to reconstruct the original image starting from the most meaningful content output by the compression network 210. For example, the reconstruction network 212 uses the most meaningful content to select locations to add traffic lights that are likely to be consistent with the semantics of the road scene. The first discriminator 206 and the second discriminator 208 are configured to compare the reconstructed image generated by the generator 204 to the original image to determine whether the reconstructed image is real or not real. The first discriminator 206 and the second discriminator 208 compare different proportions of the reconstructed image. For example, the first discriminator 206 determines whether the final reconstructed image generated by the generator 204 is real or not real. Moreover, the second discriminator 208 determines whether earlier stages of the reconstructed image generated by the generator 204 are real or not real. Based on the determinations by the first discriminator 206 and the second discriminator 208, the generator 204 is configured to adjust parameters within the compression network 210 and the reconstruction network 212 that are used to determine the locations to add traffic lights that are likely to be consistent with the semantics of the road scene.

[0026] After multiple iterations of the training described above, the generator 204 is configured to consistently generate reconstructed images that are determined to be real by the first discriminator 206, the second discriminator 208, or both. As a result of the training, the GAN model 200 determines a distribution of the likelihood that any location in a real road scene image can contain a traffic light based on the semantics of the road scene. The determined distribution is used to generate augmented road scene images that are consistent with the semantics of the road scene, as will be described in more detail below. Moreover, the determined distribution includes anchor centers for sampling to detect small and occluded traffic lights.

[0027] Figure 3A and Figure 3B is a flowchart of one example of a method 300 for generating augmented training data using a GAN (e.g., using the GAN model 200). In some implementations, the network trainer 114 includes computer-readable instructions that cause the electronic processor 104 to perform all of the blocks (or any combination) of the method 300 described below. At block 302, a plurality of real-world road scene images is received (e.g., by the electronic processor 104). In some implementations, all (or any portion) of the plurality of real-world road scene images is stored in the memory 106. For example, the plurality of real-world road scene images can be included in the image data 118 stored in the memory 106. Alternatively or additionally, the computer 102 receives all (or any portion) of the plurality of real-world road scene images from an external source via the input / output interface 108. For example, the computer 102 can receive the plurality of real-world road scene images from a separate computer, server, database, etc.

[0028] At block 304, a subset of real-world road scene images is selected (e.g., by the electronic processor 104) from the plurality of real-world road scene images. The subset includes any portion of the plurality of real-world road scene images. For example, the subset can include less than the entire plurality of real-world road scene images or include the entire plurality of real-world road scene images. In some implementations, the number of images selected to be included in the subset is determined based on a predetermined percentage of the total number of images included in the plurality of real-world road scene images. For example, the subset can include half of the total number of images included in the plurality of real-world road scene images. In some implementations, the number of images selected to be included in the subset is determined based on a predetermined number. As a first example, when the predetermined number is 500 images and the plurality of real-world road scene images includes more than 500 images, the subset includes 500 images. As a second example, when the predetermined number is 500 images and the plurality of real-world road scene images includes 500 images or less, the subset includes the entire plurality of real-world road scene images.

[0029] At block 306, a real-world road scene image is selected from the subset of real-world road scene images. For example, the electronic processor 104 selects a first image included in the subset of real-world road scene images. At block 308, a traffic light is removed from the selected real-world road scene image to generate a repaired image. For example, the image repairer 202 removes the traffic light from the selected real-world road scene image to generate a repaired image. In some implementations, the image repairer 202 sets each pixel in the selected real-world road scene image that contains a traffic light to a predetermined value (e.g., zero). At block 310, the GAN model is trained using the repaired image. For example, the GAN model is trained using the repaired image as described above with respect to the method 200. Figure 2The repaired image trains the GAN model from scratch.

[0030] At block 312, a distribution indicating a likelihood that each location in the selected real road scene image can contain a traffic light is determined using the GAN model. For example, the electronic processor 104 determines the distribution using the GAN model 200 described above with respect to Figure 2 At block 314, a location in the selected real road scene image is selected by sampling the distribution. For example, the electronic processor 104 samples the distribution to select a location with a high likelihood of containing a traffic light. Turning to Figure 3B At block 316, a traffic light image is superimposed onto the selected real road scene image at the selected location to generate an enhanced road scene image.

[0031] At block 318, the electronic processor 104 determines whether each image in the subset of real road scene images has been processed, e.g., to generate an enhanced image. When the subset of real road scene images has not all been processed, the method 300 returns to block 306 in Figure 3A to select another one of the subset of real road scene images. Alternatively, when each image in the subset of real road scene images has been processed, a plurality of enhanced road scene images have been generated, and the method 300 proceeds to block 320. At block 320, the pre-trained deep neural network model is trained using the plurality of real road scene images and the plurality of enhanced road scene images to generate a newly trained deep neural network model. For example, the deep neural network model 116 stored in the memory 106 is trained to detect traffic lights in image data using the plurality of real road scene images and the plurality of enhanced road scene images. In some implementations, the pre-trained deep neural network model is an untrained deep neural network model. In other implementations, the pre-trained deep neural network model has been previously trained, e.g., with image data not included in the plurality of real road scene images.

[0032] In some implementations, the enhanced road scene images are generated based on global semantics of each image in the dataset. For example, the enhanced road scene images are generated using prior knowledge of traffic light placements empirically estimated from the plurality of real road scene images. Figure 4A and Figure 4B is a flowchart of one example of a method 400 for generating enhanced training data using prior knowledge. In some implementations, the network trainer 114 includes computer-readable instructions that cause the electronic processor 104 to perform all of the blocks (or any combination) of the method 400 described below. At block 402, a plurality of real road scene images is received. For example, the electronic processor 104 receives a plurality of real road scene images, as described above with respect to Figure 3AAs described in box 302. In box 404, a subset of real-world road scene images is selected from a plurality of real-world road scene images. For example, electronic processor 104 selects a subset of real-world road scene images from a plurality of real-world road scene images, as described above regarding... Figure 3A As described in box 304. In box 406, a real-world road scene image is selected from a subset of real-world road scene images. For example, electronic processor 104 selects a first image included in the subset of real-world road scene images.

[0033] In box 408, a first distribution indicating the probability that each location in the selected real-world road scene may contain a traffic light is determined based on the relative frequency of traffic light occurrences at each location across all the multiple real-world road scene images. For example, when the multiple real-world road scene images comprise 497 images, the first distribution may, among other things, specifically indicate that 329 images contain a traffic light at a first location, and 53 images contain a traffic light at a second location different from the first location. In box 410, locations in the selected real-world road scene images are selected by sampling the first distribution. For example, electronic processor 104 samples the first distribution to select locations with a high probability of containing a traffic light.

[0034] In box 412, a second distribution indicating possible traffic light sizes at a selected location is determined based on the relative frequency of traffic light sizes at that location within any of a plurality of real-world road scene images. For example, when the plurality of real-world road scene images comprises 497 images and 329 images include traffic lights at the selected location, the second distribution could indicate that 211 traffic lights at the selected location have a first set of sizes. The second distribution could also indicate that 99 traffic lights at the selected location have a second set of sizes different from the first set of sizes. The second distribution could also indicate that 19 traffic lights at the selected location have a third set of sizes different from both the first and second set of sizes. Traffic light sizes include height, width, or both. Go to Figure 4B In box 414, the size of the traffic light image to be overlaid onto the selected real-world road scene image is selected by sampling the second distribution. For example, the electronic processor 104 samples the second distribution to select the most common height and width for traffic lights among multiple real-world road scene images.

[0035] In box 416, a traffic light image of the selected size is overlaid onto a selected real-world road scene image at the selected location to generate an enhanced road scene image. In box 418, the electronic processor 104 determines whether each image in a subset of the real-world road scene images has been processed, for example, to generate the enhanced image. When not all of the subset of the real-world road scene images has been processed, method 400 returns to... Figure 4AIn box 406, another image is selected from the subset of real-world road scene images. Alternatively, when each image in the subset of real-world road scene images has been processed, multiple augmented road scene images have been generated, and method 400 proceeds to box 420. In box 420, a pre-trained deep neural network model is trained using the multiple real-world road scene images and the multiple augmented road scene images to generate a newly trained deep neural network model. For example, deep neural network model 116 stored in memory 106 is trained to detect traffic lights in the image data using the multiple real-world road scene images and the multiple augmented road scene images.

[0036] In some implementations, method 400 further includes using Poisson mixture to filter each of the multiple enhanced road scene images. In some implementations, Poisson mixture is used to filter each of the multiple enhanced road scene images before training a pre-trained deep neural network model. In some implementations, method 400 does not include boxes 412 and 414, and method 400 proceeds to box 416 after box 410. For example, in some implementations, the size of the traffic light image used for overlay is a predetermined value.

[0037] Figure 5 This is a flowchart of an example of a method 500 for generating augmented data to train a deep neural network to detect traffic lights in image data. In some implementations, the network trainer 114 includes computer-readable instructions that cause the electronic processor 104 to execute all (or any combination thereof) the boxes of method 500 described below. In box 502, multiple real-world road scene images are received. For example, the electronic processor 104 receives multiple real-world road scene images, as described above. Figure 3A As described in box 302. In box 504, a subset of real-world road scene images is selected from a plurality of real-world road scene images. For example, electronic processor 104 selects a subset of real-world road scene images from a plurality of real-world road scene images, as described above regarding... Figure 3A As described in box 304. In box 506, a real-world road scene image is selected from a subset of real-world road scene images. For example, electronic processor 104 selects a first image included in the subset of real-world road scene images. In box 508, (e.g., by electronic processor 104) a distribution indicating the probability that each location in the selected real-world road scene may contain a traffic light is determined. In some implementations, for example, as described above regarding... Figure 3A As described in box 312, a GAN is used to determine the distribution. In other implementations, for example, as mentioned above... Figure 4AAs described in box 408, a distribution is determined based on the relative frequency of traffic light occurrences at each location in all multiple real-world road scene images. In box 510, locations in selected real-world road scene images are selected by sampling the distribution. For example, electronic processor 104 samples the distribution to select locations with a high probability of containing traffic lights. In box 514, traffic light images are overlaid on the selected real-world road scene images at the selected locations to generate enhanced road scene images. In box 514, electronic processor 104 determines whether each image in a subset of real-world road scene images has been processed, for example, to generate an enhanced image. When not all subsets of real-world road scene images have been processed, method 500 returns to box 506 to select another subset of real-world road scene images. Alternatively, when each image in a subset of real-world road scene images has been processed, multiple enhanced road scene images have been generated, and method 500 proceeds to box 516. In box 516, a pre-trained deep neural network model is trained using the multiple real-world road scene images and the multiple enhanced road scene images to generate a newly trained deep neural network model. For example, a deep neural network model 116 stored in memory 106 is trained to detect traffic lights in image data using multiple real-world road scene images and multiple augmented road scene images.

[0038] Figure 6 Examples of enhanced road scene images generated using different enhancement techniques are included. The images in the column labeled "Random Aug" are examples of enhanced images generated by inserting traffic light images at random locations with random width and height configurations. The images in the column labeled "Prior Aug" are examples of enhanced images generated using prior knowledge, for example, as described above regarding method 400. The images in the column labeled "Poisson Aug" are examples of enhanced images generated by filtering the images in the "Prior Aug" column using Poisson mixture filtering. The images in the column labeled "GAN Aug1" are examples of enhanced images generated using GANs, for example, as described above regarding method 300.

[0039] Figure 7This is a graph illustrating the average accuracy of a deep neural network model detecting traffic lights, trained using training data with varying degrees of realism. The "Realistic Data" row in the graph represents the average accuracy when the deep neural network model is trained using 500 or more real-world road scene images. The "Random Augmentation" row represents the average accuracy when the deep neural network model is trained using 500 real-world road scene images and different numbers of augmented road scene images generated by inserting traffic light images at random locations with random width and height configurations. The "Prior Knowledge" row represents the average accuracy when the deep neural network model is trained using 500 real-world road scene images and different numbers of augmented road scene images generated using prior knowledge, as described above with respect to method 400. The "Poisson Augmentation" row represents the average accuracy when the deep neural network model is trained using 500 real-world road scene images and different numbers of augmented road scene images generated using prior knowledge and Poisson filtering. For example, using the method 400 described above to generate augmented road scene images, the method 400 has an additional step of using Poisson mixture to filter each of a plurality of augmented road scene images before training a deep neural network model. The "GAN Augmentation" row in the graph represents the average accuracy when training a deep neural network model using 500 real-world road scene images and, for example, different numbers of augmented road scene images generated using GAN as described above with respect to method 300.

[0040] Figure 7 The horizontal axis of the chart indicates the total number of training images used to train the deep neural network model. The "Based" label on the horizontal axis indicates when 500 real-world road scene images were used to train the deep neural network model. For the "Real-world Data" row, the "0.25" label on the horizontal axis indicates that 625 real-world road scene images were used to train the deep neural network model. For other rows, the "0.25" label on the horizontal axis indicates that 500 real-world road scene images and 125 augmented road scene images generated using the corresponding data augmentation techniques described above were used to train the deep neural network model. For the "Real-world Data" row, the "1" label on the horizontal axis indicates that 1000 real-world road scene images were used to train the deep neural network model. For other rows, the "1" label on the horizontal axis indicates that 500 real-world road scene images and 500 augmented road scene images generated using the corresponding data augmentation techniques described above were used to train the deep neural network model. For the "Real-world Data" row, the "10" label on the horizontal axis indicates that 5500 real-world road scene images were used to train the deep neural network model. For the other rows, the "10" label on the horizontal axis indicates that the deep neural network model was trained using 500 real-world road scene images and 5,000 augmented road scene images generated using the corresponding data augmentation techniques described above.

[0041] Figure 7 The charts in the diagram illustrate that both prior knowledge augmentation and Poisson augmentation achieve similar performance improvements in traffic light detection compared to using additional real-world data. Figure 7 The charts also illustrate that GAN enhancements achieve similar or greater improvements in traffic light detection performance compared to using additional real-world data. Figure 7 The diagrams further illustrate that prior knowledge augmentation, Poisson augmentation, and GAN augmentation require a much larger amount of data to achieve similar performance improvements as using additional real-world data. However, obtaining additional real-world data is logically much more difficult than generating more augmentation data using the augmentation techniques described in this paper. Figure 7 The charts also illustrate that random augmentation with semantic inconsistencies does not achieve significant performance improvements in traffic light detection.

[0042] Table 1 illustrates a comparison of the sampling efficiency of prior knowledge enhancement and GAN enhancement in detecting traffic lights that other models have not detected.

[0043] Table 1 Comparison of Sampling Efficiency

[0044]

[0045] The various aspects of this disclosure can take any one or more of the following exemplary configurations.

[0046] EEE (1) A method for generating augmented data to train a deep neural network to detect traffic lights in image data, the method comprising the steps of: (a) receiving a plurality of real-world road scene images; (b) selecting a subset of real-world road scene images from the plurality of real-world road scene images; (c) selecting real-world road scene images from the subset of real-world road scene images; (d) determining a distribution indicating the probability that each location in the selected real-world road scene images may contain a traffic light; (e) selecting locations in the selected real-world road scene images by sampling the distribution; (f) overlaying traffic light images onto the selected real-world road scene images at the selected locations to generate augmented road scene images; (g) repeating steps (b) to (f) for each real-world road scene image in the subset of real-world road scene images to generate a plurality of augmented road scene images; and (h) using the plurality of real-world road scene images and the plurality of augmented road scene images to train a pre-trained deep neural network model to generate a newly trained deep neural network model.

[0047] EEE(2) is the method according to EEE(1), wherein step (d) includes: removing traffic lights from a selected real-world road scene image to generate a repaired image, training a generative adversarial network (GAN) model using the repaired image, and determining a distribution using the GAN model.

[0048] EEE (3) is the method according to EEE (1) or EEE (2), wherein step (d) includes determining the distribution based on the relative frequency of traffic light occurrences at each location in all of the plurality of real-world road scene images.

[0049] EEE (4) further includes the step of using Poisson mixing to filter each of the plurality of enhanced road scene images according to any one of EEE (1) to EEE (3).

[0050] EEE (5) is the method according to EEE (4), wherein, before training a pre-trained deep neural network model, Poisson mixture is used to filter each of the plurality of enhanced road scene images.

[0051] EEE (6) according to any one of EEE (1) to EEE (5), wherein the distribution is a first distribution, and wherein step (f) comprises: determining a second distribution indicating possible traffic light sizes at a selected location based on the relative frequency of traffic light sizes at a selected location in any one of the plurality of real-world road scene images, selecting a size for a traffic light image by sampling the second distribution, and overlaying the traffic light image onto the selected real-world road scene image at the selected location to generate an enhanced road scene image, wherein the traffic light image includes the selected size.

[0052] EEE (7) is the method described in EEE (6), wherein the selected dimensions include the height of the traffic light image and the width of the traffic light image.

[0053] EEE (8) A system for generating augmented data to train a deep neural network to detect traffic lights in image data, the system comprising: an electronic processor; and a memory storing computer-readable instructions, which, when executed by the electronic processor, cause the electronic processor to perform operations including: (a) receiving a plurality of real-world road scene images; (b) selecting a subset of the real-world road scene images from the plurality of real-world road scene images; (c) selecting real-world road scene images from the subset of real-world road scene images; (d) determining a distribution indicating the probability that a traffic light may be present at each location in the selected real-world road scene images; (e) selecting a location in the selected real-world road scene images by sampling the distribution; (f) overlaying a traffic light image onto the selected real-world road scene image at the selected location to generate an augmented road scene image; (g) repeating operations (b) to (f) for each real-world road scene image in the subset of real-world road scene images to generate a plurality of augmented road scene images; and (h) using the plurality of real-world road scene images and the plurality of augmented road scene images to train a pre-trained deep neural network model to generate a newly trained deep neural network model.

[0054] EEE (9) According to the system described in EEE (8), wherein the operation (d) includes: removing traffic lights from a selected real-world road scene image to generate a repaired image, training a generative adversarial network (GAN) model using the repaired image, and determining the distribution using the GAN model.

[0055] EEE (10) is a system based on EEE (8) or EEE (9), wherein the operation (d) includes determining the distribution based on the relative frequency of traffic light occurrences at each location in all of the plurality of real-world road scene images.

[0056] The system according to any one of EEE (8) to EEE (10), wherein the operation further comprises using Poisson blending to filter each of the plurality of enhanced road scene images.

[0057] EEE (12) is based on the system described in EEE (11), wherein Poisson mixing is used to filter each of the plurality of enhanced road scene images before training a pre-trained deep neural network model.

[0058] EEE (13) The system according to any one of EEE (8) to EEE (12), wherein the distribution is a first distribution, and wherein the operation (f) comprises: determining a second distribution indicating possible traffic light sizes at a selected location based on the relative frequency of the sizes of traffic lights at a selected location in any one of the plurality of real-world road scene images; selecting a size for a traffic light image by sampling the second distribution; and overlaying the traffic light image onto a selected real-world road scene image at the selected location to generate an enhanced road scene image, wherein the traffic light image includes the selected size.

[0059] EEE (14) is based on the system described in EEE (13), wherein the selected dimensions include the height of the traffic light image and the width of the traffic light image.

[0060] EEE (15) A non-transitory computer-readable medium storing computer-readable instructions that, when executed by an electronic processor of a computer, cause the computer to perform operations, the operations comprising: (a) receiving a plurality of real-world road scene images; (b) selecting a subset of the real-world road scene images from the plurality of real-world road scene images; (c) selecting real-world road scene images from the subset of real-world road scene images; (d) determining a distribution indicating the probability that a traffic light may be present at each location in the selected real-world road scene images; (e) selecting a location in the selected real-world road scene images by sampling the distribution; (f) overlaying a traffic light image onto the selected real-world road scene image at the selected location to generate an enhanced road scene image; (g) repeating operations (b) to (f) for each real-world road scene image in the subset of real-world road scene images to generate a plurality of enhanced road scene images; and (h) using the plurality of real-world road scene images and the plurality of enhanced road scene images to train a pre-trained deep neural network model to generate a newly trained deep neural network model.

[0061] EEE (16) according to the non-transitory computer-readable medium of EEE (15), wherein the operation (d) includes: removing traffic lights from a selected road scene image to generate a repaired image, training a generative adversarial network (GAN) model using the repaired image, and determining the distribution based on the GAN model.

[0062] EEE (17) according to the non-transitory computer-readable medium of EEE (15) or EEE (16), wherein the operation (d) includes determining the distribution based on the relative frequency of traffic light occurrences at each location in all of the plurality of real-world road scene images.

[0063] EEE (18) A non-transitory computer-readable medium according to any one of EEE (15) to EEE (17), wherein the operation further comprises using Poisson mixing to filter each of the plurality of enhanced road scene images.

[0064] EEE (19) is a non-transitory computer-readable medium according to EEE (18), wherein each of the plurality of enhanced road scene images is filtered using Poisson mixing before training a pre-trained deep neural network model.

[0065] EEE (20) a non-transitory computer-readable medium according to any one of EEE (15) to EEE (19), wherein the distribution is a first distribution, and wherein the operation (f) comprises: determining a second distribution indicating possible traffic light sizes at a selected location based on the relative frequency of traffic light sizes at a selected location in any of the plurality of real-world road scene images; selecting a size for a traffic light image by sampling the second distribution; and overlaying the traffic light image onto a selected real-world road scene image at the selected location to generate an enhanced road scene image, wherein the traffic light image includes the selected size.

[0066] Therefore, this disclosure provides, among other things, methods, systems, and non-transitory computer-readable media for generating augmented data to train deep neural networks to detect traffic lights in image data. Various features and advantages are set forth in the following claims.

[0067] Machine learning generally refers to the ability of a computer program to learn without being explicitly programmed. In some implementations, the computer program (e.g., a learning engine) is configured to construct algorithms based on inputs. Supervised learning involves presenting a computer program with example inputs and their expected outputs. The computer program is configured to learn general rules that map inputs to outputs from its received training data. Example machine learning engines include decision tree learning, association rule learning, artificial neural networks, classifiers, inductive logic programming, support vector machines, clustering, Bayesian networks, reinforcement learning, representation learning, similarity and metric learning, sparse dictionary learning, and genetic algorithms. Using one or more of these methods, a computer program can ingest, parse, and understand data, and progressively refine algorithms for data analysis.

[0068] Specific embodiments have been described in the foregoing specification. However, those skilled in the art will appreciate that various modifications and alterations can be made without departing from the scope of the invention as set forth in the following claims. Therefore, the specification and figures are to be considered in an illustrative sense rather than a restrictive sense, and all such modifications are intended to be included within the scope of this teaching.

[0069] Benefits, advantages, solutions to problems, and any element(s) that may lead to or make any benefit, advantage, or solution appear or become more significant shall not be construed as a critical, claimed, or substantial feature or element of any or all of the claims. The invention is defined solely by the appended claims—including any modifications made during the pending period of this application and all equivalents of those claims as published.

[0070] Furthermore, in this document, relational terms such as first and second, top and bottom, etc., may be used only to distinguish one entity or action from another, and do not necessarily require or imply any actual such relationship or order between such entities or actions. The terms “comprising,” “containing,” “having,” “including,” “comprise,” “containing,” or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that includes, has, comprises, or contains a list of elements includes not only those elements but may also include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element following “comprising…a,” “having…a,” “including…a,” or “containing…a” does not exclude the presence of additional identical elements in a process, method, article, or apparatus that includes, has, comprises, or contains that element, unless otherwise expressly stated herein. The terms “a” and “an” are defined as one or more unless otherwise expressly stated herein. The terms “substantially,” “essentially,” “approximately,” “about,” or any other version thereof are defined as close to what is understood by one of ordinary skill in the art, and in one non-limiting embodiment, the term is defined as within 10%, in another within 5%, in yet another within 1%, and in yet another within 0.5%. The term “coupling,” as used herein, is defined as a connection, although not necessarily a direct connection and not necessarily a mechanical connection. A device or structure that is “configured” in a certain way is at least configured in this way, but may also be configured in ways not listed.

[0071] An abstract is provided to allow the reader to quickly understand the nature of the technical disclosure. The abstract is submitted under the understanding that it will not be used to interpret or limit the scope or meaning of the claims. Furthermore, as can be seen from the foregoing detailed description, various features have been grouped together in various embodiments for the purpose of simplifying this disclosure. This method of disclosure should not be construed as reflecting an intention to claim more features than expressly recited in each claim. Rather, as reflected in the following claims, the inventive subject matter lies in fewer than all features of a single disclosed embodiment. Therefore, the following claims are thus incorporated into the detailed description, wherein each claim is, on its own, an independent claim.

Claims

1. A method for generating augmented data to train a deep neural network to detect traffic lights in image data, the method comprising the following steps: (a) Receive multiple images of real-world road scenes; (b) Selecting a subset of the real-world road scene images from the plurality of real-world road scene images; (c) Select real-world road scene images from a subset of real-world road scene images; (d) Determine a distribution that indicates the likelihood that each location in the selected real-world road scene image may contain a traffic light; (e) Selecting a location in a chosen real-world road scene image by sampling the distribution; (f) Overlay the traffic light image onto the selected real-world road scene image at the selected location to generate an enhanced road scene image; (g) Repeat steps (b) through (f) for each real road scene image in a subset of real road scene images to generate multiple enhanced road scene images; and (h) Using the plurality of real-world road scene images and the plurality of augmented road scene images to train a pre-trained deep neural network model to generate a newly trained deep neural network model. The step (d) mentioned therein includes: Remove traffic lights from a selected real-world road scene image to generate a repaired image. Using repaired images to train a generative adversarial network (GAN) model, and The distribution was determined using a GAN model.

2. The method of claim 1, further comprising the step of using Poisson mixing to filter each of the plurality of enhanced road scene images.

3. The method according to claim 2, wherein, Before training a pre-trained deep neural network model, Poisson mixing is used to filter each of the multiple enhanced road scene images.

4. The method of claim 1, wherein the distribution is a first distribution, and wherein step (f) comprises: Based on the relative frequency of traffic light sizes at selected locations in any of the plurality of real-world road scene images, a second distribution indicating possible traffic light sizes at the selected locations is determined. The size for the traffic light image is selected by sampling the second distribution, and The traffic light image is overlaid onto the selected real-world road scene image at the selected location to generate an enhanced road scene image, wherein the traffic light image includes the selected size.

5. The method of claim 4, wherein the selected dimensions include the height of the traffic light image and the width of the traffic light image.

6. A system for generating augmented data to train a deep neural network to detect traffic lights in image data, the system comprising: Electronic processor; and A memory storing computer-readable instructions, which, when executed by an electronic processor, cause the electronic processor to perform operations, including: (a) Receive multiple images of real-world road scenes, (b) Selecting a subset of the real-world road scene images from the plurality of real-world road scene images. (c) Select real-world road scene images from a subset of real-world road scene images. (d) Determine a distribution that indicates the likelihood that a traffic light may be present at each location in a selected image of a real-world road scene. (e) Selecting a location in a selected real-world road scene image by sampling the distribution. (f) Overlay the traffic light image onto the selected real-world road scene image at the selected location to generate an enhanced road scene image. (g) Repeat operations (b) to (f) for each real-world road scene image in a subset of the real-world road scene images to generate multiple enhanced road scene images, and (h) Using the plurality of real-world road scene images and the plurality of augmented road scene images to train a pre-trained deep neural network model to generate a newly trained deep neural network model. The operation (d) mentioned above includes: Remove traffic lights from a selected real-world road scene image to generate a repaired image. The restored image is used to train and generate an adversarial network GAN model, and The distribution was determined using a GAN model.

7. The system of claim 6, wherein the operation further comprises using Poisson mixing to filter each of the plurality of enhanced road scene images.

8. The system according to claim 7, wherein, Before training a pre-trained deep neural network model, Poisson mixing is used to filter each of the multiple enhanced road scene images.

9. The system of claim 6, wherein the distribution is a first distribution, and wherein the operation (f) comprises: Based on the relative frequency of traffic light sizes at selected locations in any of the plurality of real-world road scene images, a second distribution indicating possible traffic light sizes at the selected locations is determined. The size for the traffic light image is selected by sampling the second distribution, and The traffic light image is overlaid onto the selected real-world road scene image at the selected location to generate an enhanced road scene image, wherein the traffic light image includes the selected size.

10. The system of claim 9, wherein the selected dimensions include the height of the traffic light image and the width of the traffic light image.

11. A non-transitory computer-readable medium storing computer-readable instructions, said computer-readable instructions, when executed by a computer's electronic processor, causing the computer to perform operations, said operations including: (a) Receive multiple images of real-world road scenes; (b) Selecting a subset of the real-world road scene images from the plurality of real-world road scene images; (c) Select real-world road scene images from a subset of real-world road scene images; (d) Determine a distribution that indicates the likelihood that a traffic light may be present at each location in a selected image of a real-world road scene; (e) Selecting a location in a selected real-world road scene image by sampling the distribution; (f) Overlay the traffic light image onto the selected real-world road scene image at the selected location to generate an enhanced road scene image; (g) Repeat operations (b) to (f) for each real road scene image in a subset of real road scene images to generate multiple enhanced road scene images; and (h) Using the plurality of real-world road scene images and the plurality of augmented road scene images to train a pre-trained deep neural network model to generate a newly trained deep neural network model. The operation (d) mentioned above includes: Remove traffic lights from the selected road scene image to generate a repaired image. Using repaired images to train a generative adversarial network (GAN) model, and The distribution is determined based on the GAN model.

12. The non-transitory computer-readable medium of claim 11, wherein the operation further comprises using Poisson mixing to filter each of the plurality of enhanced road scene images.

13. The non-transitory computer-readable medium according to claim 12, wherein, Before training a pre-trained deep neural network model, Poisson mixing is used to filter each of the multiple enhanced road scene images.

14. The non-transitory computer-readable medium of claim 12, wherein the distribution is a first distribution, and wherein the operation (f) comprises: Based on the relative frequency of traffic light sizes at selected locations in any of the plurality of real-world road scene images, a second distribution indicating possible traffic light sizes at the selected locations is determined. The size for the traffic light image is selected by sampling the second distribution, and The traffic light image is overlaid onto the selected real-world road scene image at the selected location to generate an enhanced road scene image, wherein the traffic light image includes the selected size.

Images