Method for creating a dataset of an image processing apparatus, and image processing apparatus

Abstract

To provide a dataset generation method for an image processing apparatus and the image processing apparatus which allow for generating a dataset close to an actual image from a simulation image (virtual image) at a low cost.SOLUTION: A dataset generation method for an image processing apparatus includes steps of preparing an actual vehicle color image, processing the actual vehicle color image to generate an actual vehicle black-and-white image, generating colorizing AI for restoring the actual vehicle color image from the actual vehicle black-and-white image, generating a simulation color image, converting the simulation color image to a simulation black-and-white image corresponding to the actual vehicle black-and-white image, converting the simulation black-and-white image to an object detection training image by the colorizing AI, and extracting annotation information of the object detection training image.SELECTED DRAWING: Figure 6

Description

Technical Field

[0001] The present invention relates to a method for creating a dataset of an image processing apparatus and an image processing apparatus.

Background Art

[0002] Conventionally, a method for creating a dataset of an image processing apparatus and an image processing apparatus have been proposed (see, for example, Patent Documents 1 to 5).

[0003] Patent Document 1 discloses an image processing apparatus that efficiently collects a dataset for machine learning. In order to perform colorization processing for each object in a scanned image, an object included in a page image corresponding to input page-based image data is detected. Further, Patent Document 1 describes that when the detected object is a monochrome image, colorization processing for converting the monochrome image into a color image is executed for each object, and when the detected object is a color image, the image data of the color image and the image data converted into a monochrome image are stored as learning samples for parameters related to the colorization processing.

[0004] Patent Document 2 discloses an image processing apparatus that efficiently collects learning samples used for machine learning in colorization processing and performs machine learning. It is described that it is determined whether the input page-based image data is a color image or a monochrome image. Further, Patent Document 2 describes that when the input image data is determined to be a monochrome image, colorization processing for converting the monochrome image into a color image is executed, and when the input image data is determined to be a color image, it is determined whether to use the image data as a learning sample based on the colorization result obtained by once converting the image data of the color image into a monochrome image and then converting it back into a color image.

[0005] Patent Documents 3 and 4 disclose a discriminator training method for stabilizing the training of a data discriminator (discriminator) in a Generative Adversarial Networks (GAN). The GANs disclosed in Patent Documents 3 and 4 have been widely studied in recent years as a framework for generative models and are applied to the creation of various datasets as described in Patent Documents 1 and 2. Patent Documents 3 and 4 also describe a discriminator training method for training a data discriminator equipped with a neural network model that discriminates between correct data and fake data. The correct data is input to the discriminator to obtain a first prediction result, the fake data is input to the discriminator to obtain a second prediction result, an error is calculated based on these first and second prediction results, and the weight matrix is updated based on the error and the singular values of the weight matrices of the respective layers of the neural network model.

[0006] Patent Document 5 discloses a learning method for generating training data obtained from virtual data in a virtual world. Patent Document 5 describes that, for the purpose of creating an image recognition AI used in an autonomous vehicle, a learning dataset such as training data and teacher data (annotations) is generated from simulation images (virtual images), and GANs as described in Patent Documents 3 and 4 are used to reduce the difference between the simulation images and real-world images (real images).

[0007] As described in Patent Documents 1 to 5, there is a technique for creating a learning dataset closer to real images from simulation images using GANs in order to reduce the preparation cost of the learning dataset. For example, in the technique described in Patent Document 5, (a) a feature map is generated from a simulation image, (b) a transformed image is generated from the feature map, (c) a category score vector in the transformed image is generated, and (d) the parameters of the generating CNN are learned with reference to the category score vector and the corresponding real image, and as a result, the simulation image is transformed into a more realistic image.

Prior Art Documents

Patent Documents

[0008]

Patent Document 1

Patent Document 2

Patent Document 3

Patent Document 4

Patent Document 5

Summary of the Invention

Problems to be Solved by the Invention

[0009] However, as described above, when using the technique of creating a learning dataset closer to real images from simulation images using GAN, since it is necessary to generate a category score vector even in real images, it takes an enormous amount of time to build an AI (Artificial Intelligence) model used for determining the category score vector, and there is the disadvantage that the cost increases accordingly. For this reason, a method for creating a dataset for an image processing apparatus capable of creating a dataset close to real images from simulation images at low cost, and an image processing apparatus are desired.

[0010] The present invention has been made to solve the above problems, and an object thereof is to provide a method for creating a dataset for an image processing apparatus and an image processing apparatus capable of creating a dataset close to real images from simulation images (virtual images) at low cost.

Means for Solving the Problems

[0011] To achieve the above object, the present invention is configured as follows.

[0012] (1) The method for creating a data set of an image processing apparatus according to the present invention includes a step of preparing a real image, a step of creating a processed image by processing the real image, a step of creating a learned model for restoring the real image from the processed image using the real image and the processed image, a step of creating a first virtual image, a step of converting the first virtual image into a second virtual image corresponding to the processed image, a step of converting the second virtual image into a third virtual image by the learned model, and a step of extracting annotation information of the third virtual image.

[0013] According to the above configuration, after converting the first virtual image as a simulation image into a second virtual image, and then converting the second virtual image into a third virtual image by the learned model, a learning data set close to the real image can be created from the first virtual image. As a result, it is possible to create a data set close to the real image from the first virtual image as a simulation image at low cost without taking time to construct the learned model.

[0014] (2) In the method for creating a data set of an image processing apparatus according to the present invention, preferably, the learned model is created by a GAN (Generative Adversarial Networks) including a generator, a discriminator, and a prediction network. With this configuration, the second virtual image can be converted into a high-quality third virtual image by the learned model created by the GAN.

[0015] (3) In the method for creating a data set of an image processing apparatus according to the present invention, preferably, the real image is an image captured by an imaging device provided on a vehicle. With this configuration, since the learned model is created using the image captured by the imaging device provided on the vehicle as a learning image, a learned model reflecting the characteristics (lens, sensor, color tone, etc.) of the imaging device provided on the vehicle can be created.

[0016] (4) In this case, preferably, the step of preparing the real image includes the step of preparing the real color image captured by the imaging device, and the step of creating the processed image includes the step of creating the black-and-white processed image by converting the color real image into black-and-white. The learned model is an AI (Artificial Intelligence) that restores the color real image from the black-and-white processed image. With this configuration, when converting the second virtual image into the third virtual image using the learned model (AI), the second virtual image can be converted into the third virtual image in which the characteristics of the imaging device (such as lens, sensor, color tone, etc.) are reproduced.

[0017] (5) The image processing apparatus according to the present invention includes a processed image creation unit that creates a processed image by processing the prepared real image, a learned model creation unit that creates a learned model that restores the real image from the processed image using the real image and the processed image, a first virtual image creation unit that creates a first virtual image, a second virtual image conversion unit that converts the first virtual image into a second virtual image corresponding to the processed image, a third virtual image conversion unit that converts the second virtual image into a third virtual image by the learned model, and an annotation information extraction unit that extracts annotation information of the third virtual image.

[0018] According to the above configuration, after converting the first virtual image as a simulation image into the second virtual image in the second virtual image conversion unit, the second virtual image is converted into the third virtual image by the learned model in the third virtual image conversion unit, so that a learning dataset close to the real image can be created from the first virtual image. As a result, it is possible to create a dataset close to the real image from the first virtual image as a simulation image at low cost without taking time to build the learned model.

Effects of the Invention

[0019] According to an aspect of the present invention, it is possible to provide a method for creating a dataset of an image processing apparatus that can create a dataset close to a real image from a simulation image (virtual image) at low cost, and the image processing apparatus.

Brief Description of Drawings

[0020]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Embodiments for Carrying Out the Invention

[0021] Hereinafter, with reference to the accompanying drawings, an image processing apparatus and a system including a vehicle according to an embodiment of the present invention will be described. These drawings are schematic diagrams and are not necessarily drawn at an accurate ratio of size. Also, in the drawings, the same components are denoted by the same reference numerals.

[0022] With reference to FIGS. 1 to 5, an image processing apparatus and a system including a vehicle according to the present embodiment will be described.

[0023] As shown in FIG. 1, the system according to this embodiment mainly includes a vehicle 10 and an image processing apparatus 20. The vehicle 10 and the image processing apparatus 20 are connected to each other so as to be able to communicate information by wireless connection or wired connection.

[0024] (Vehicle) The vehicle 10 mainly includes an ECU (Engine Control Unit) 11, an object detector 12, and a network I / F 13. The ECU 11 is a control device that controls the entire vehicle. The object detector 12 includes a front vehicle camera, a rear vehicle camera, etc. (hereinafter sometimes referred to as "vehicle camera" or simply "camera") mounted on the vehicle 10. Note that the front vehicle camera and the rear vehicle camera are examples of the "imaging device" of the present invention. The object detector 12 is provided for detecting objects such as other vehicles, people, and obstacles. The network I / F 13 is connected to the network I / F 23 provided in the image processing apparatus 20 so as to be able to communicate information by wireless connection or wired connection.

[0025] (Image processing apparatus) The image processing apparatus 20 includes a CPU 21, a storage unit 22, a network I / F 23, and an image processing unit 24.

[0026] (CPU) The CPU 21 is a control device that controls the entire image processing apparatus.

[0027] (Storage unit) The storage unit 22 is a storage device that stores programs related to dataset creation etc. in the image processing unit 24.

[0028] (Network I / F) The network I / F 23 is connected to the network I / F 13 of the vehicle 10 so as to be able to communicate information.

[0029] (Image processing unit) Here, in the present embodiment, in the image processing unit 24, a data set of the actual vehicle color image 30a and the actual vehicle black-and-white image 30b is prepared using the actual vehicle color image 30a captured by the vehicle camera used for object detection. Also, in the image processing unit 24, camera characteristics such as the lens and sensor are considered based on these data sets, and a more realistic object detection training image 33 can be created as learning data for training the object detector 12 from the simulation color image 32a and the simulation black-and-white image 32b.

[0030] Note that the actual vehicle color image 30a is an example of the "actual image" of the present invention. The actual vehicle black-and-white image 30b is an example of the "processed image" of the present invention. The simulation color image 32a is an example of the "first virtual image" of the present invention. The simulation black-and-white image 32b is an example of the "second virtual image" of the present invention. The object detection training image 33 is an example of the "third virtual image" of the present invention.

[0031] As shown in FIG. 2, the image processing unit 24 includes an actual vehicle black-and-white image creation unit 241, a coloring AI creation unit 242, a simulation color image creation unit 243, a simulation black-and-white image conversion unit 244, an object detection training image creation unit 245, and an annotation information extraction unit 246.

[0032] Note that the actual vehicle black-and-white image creation unit 241 is an example of the "processed image creation unit" of the present invention. The coloring AI creation unit 242 is an example of the "trained model creation unit" of the present invention. The simulation color image creation unit 243 is an example of the "first virtual image creation unit" of the present invention. The simulation black-and-white image conversion unit 244 is an example of the "second virtual image conversion unit" of the present invention. The object detection training image creation unit 245 is an example of the "third virtual image conversion unit" of the present invention.

[0033] (Actual vehicle black-and-white image creation unit) As shown in FIG. 3(a), the real vehicle black-and-white image creation unit 241 creates a dataset of the real vehicle black-and-white image 30b by performing image processing on the dataset of the real vehicle color image 30a. Note that the dataset of the real vehicle color image 30a is prepared in advance by being captured by a vehicle camera.

[0034] (Coloring AI Creation Unit) As shown in FIG. 3(b), the coloring AI creation unit 242 creates the coloring AI 31. The coloring AI 31 restores the real vehicle color image 30a from the real vehicle black-and-white image 30b. The coloring AI 31 is created by a generative adversarial network (GAN) using the dataset of the real vehicle color image 30a and the dataset of the real vehicle black-and-white image 30b. The GAN includes a generator that generates real and fake data, a discriminator that discriminates the authenticity of the data generated by the generator, and a prediction network having these generator and discriminator.

[0035] Here, with reference to FIGS. 4(a) and (b), the images used when creating the coloring AI 31 will be described. When creating the object detection training image 33 (object detection learning data) from the simulation color image 32a when creating the coloring AI 31, for example, when creating an image of a rainy scene, it is desirable to increase the amount of rainy data as the teacher data for the coloring AI 31. On the other hand, when creating an image of a sunny scene, it is preferable to increase the amount of sunny data as the teacher data for the coloring AI 31. The same applies not only to the scene but also to the object. When it is desired to color the simulation black-and-white image 32b in which many cars are reflected, it is preferable to increase the amount of data in which many cars are reflected as the teacher data for the coloring AI 31.

[0036] Next, the advantages of not using the learned coloring AI will be described. When creating the object detection training image 33 (object detection learning data) using the so-called existing learned coloring AI, the characteristics of the camera used in the object detector 12 cannot be considered. On the other hand, when creating the coloring AI 31 using the actual vehicle color image 30a, etc. of the camera used in the object detector 12 as the learning image as in this embodiment, the characteristics of the camera used in the object detector 12 can be reproduced, and there is an advantage that it effectively works for the learning of the object detector 12.

[0037] (Simulation Color Image Creation Unit) The simulation color image creation unit 243 creates a simulation color image 32a (simulation source image).

[0038] (Simulation Black-and-White Image Conversion Unit) As shown in FIG. 4(a), the simulation black-and-white image conversion unit 244 converts the simulation color image 32a (simulation source image) into a simulation black-and-white image 32b corresponding to the actual vehicle black-and-white image 30b.

[0039] (Object Detection Training Image Creation Unit) As shown in FIG. 4(b), the object detection training image creation unit 245 colors the simulation black-and-white image 32b with the coloring AI 31 and converts (creates) it into an object detection training image 33 (simulation colored image).

[0040] (Annotation Information Extraction Unit) As shown in FIG. 5, the annotation information extraction unit 246 extracts the annotation information (teacher data, label) of the object detection training image 33 using the simulation function. The extracted annotation information is prepared in multiple sheets as training data (object detection learning data), and the object detector 12 in the vehicle 10 is trained using the dataset.

[0041] (Dataset Creation Method) Referring to FIGS. 1 to 6, a method for creating a dataset of the image processing apparatus 20 according to the present embodiment will be described.

[0042] As shown in FIG. 6, in step S10, a dataset of the actual vehicle color image 30a is prepared. The dataset of the actual vehicle color image 30a is prepared in advance by being captured by a vehicle camera. Also, it is desirable that the vehicle camera is an actual camera used for object detection. Then, the process proceeds to step S11.

[0043] Next, in step S11, the actual vehicle black-and-white image creation unit 241 converts the dataset of the actual vehicle color image 30a into a dataset of the actual vehicle black-and-white image 30b by performing image processing (see FIG. 3(a)). Then, the process proceeds to step S12.

[0044] Next, in step S12, the coloring AI creation unit 242 creates the coloring AI 31 by GAN using the dataset of the actual vehicle color image 30a and the dataset of the actual vehicle black-and-white image 30b (see FIG. 3(b)). Then, the process proceeds to step S13.

[0045] Next, in step S13, the simulation color image creation unit 243 creates a simulation color image 32a (original image). Then, the process proceeds to step S14.

[0046] Next, in step S14, the simulation black-and-white image conversion unit 244 converts the simulation color image 32a (simulation original image) into a simulation black-and-white image 32b corresponding to the actual vehicle black-and-white image 30b (see FIG. 4(a)). Then, the process proceeds to step S15.

[0047] Next, in step S15, the object detection training image creation unit 245 converts (creates) the object detection training image 33 (simulated colored image) by coloring the simulated black-and-white image 32b with the coloring AI 31 (see FIG. 4(b)). Then, the process proceeds to step S16.

[0048] Next, in step S16, the annotation information extraction unit 246 extracts the annotation information (teacher data, label) of the object detection training image 33 using the simulation function, prepares a plurality of them as training data (object detection learning data), creates a data set, and performs learning of the object detector 12 in the vehicle 10 (see FIG. 5). Then, the process ends.

[0049] According to the above-described embodiment, the following effects (1) to (4) can be obtained.

[0050] (1) In the data set creation method of the image processing apparatus 20 and the image processing apparatus 20 according to the above-described embodiment, the actual vehicle black-and-white image 30b was created by processing the prepared actual vehicle color image 30a. Next, the coloring AI 31 for restoring the actual vehicle color image 30a from the actual vehicle black-and-white image 30b was created. Then, the created simulated color image 32a was converted into the simulated black-and-white image 32b corresponding to the actual vehicle black-and-white image 30b. Further, the simulated black-and-white image 32b was converted into the object detection training image 33 by the coloring AI 31. Finally, the annotation information of the object detection training image 33 was extracted.

[0051] According to the above-described embodiment, after converting the simulated color image 32a into the simulated black-and-white image 32b, by converting the simulated black-and-white image 32b into the object detection training image 33 by the coloring AI 31, a training data set close to the actual vehicle color image 30a can be created from the simulated color image 32a. As a result, it is possible to create a data set close to the actual vehicle color image 30a from the simulated color image 32a at low cost without taking time to build a learned model (AI).

[0052] (2) In the method for creating a dataset of the image processing apparatus 20 according to the above embodiment, and in the image processing apparatus 20, the coloring AI 31 was created by a GAN including a generator, a discriminator, and a prediction network. Thereby, the simulation black-and-white image 32b can be converted into a high-quality object detection training image 33 by the coloring AI 31 created by the GAN.

[0053] (3) In the method for creating a dataset of the image processing apparatus 20 according to the above embodiment, and in the image processing apparatus 20, the actual vehicle color image 30a was used as an image captured by vehicle cameras such as a front camera and a rear camera provided on the vehicle. Thereby, since the coloring AI 31 is created using the actual vehicle color image 30a captured by the vehicle cameras provided on the vehicle as a learning image, it is possible to create a coloring AI 31 capable of reproducing the characteristics (lens, sensor, color tone, etc.) of the vehicle cameras.

[0054] (4) In the method for creating a dataset of the image processing apparatus 20 according to the above embodiment, and in the image processing apparatus 20, the coloring AI 31 for restoring the color actual vehicle color image 30a from the black-and-white actual vehicle black-and-white image 30b was used. Thereby, when converting the simulation black-and-white image 32b into the object detection training image 33 using the coloring AI 31, it is possible to convert the simulation black-and-white image 32b into the object detection training image 33 in which the characteristics (lens, sensor, color tone, etc.) of the vehicle cameras are reproduced.

[0055] (Modification example) The above embodiment can also be configured as follows.

[0056] In the above embodiment, an example of creating a coloring AI using an actual vehicle black-and-white image was shown, but the present invention is not limited to this. For example, it is also possible to create a coloring AI using an edge image of actual vehicle data and create an object detection training image by this coloring AI.

[0057] In the above embodiment, an example of creating a coloring AI using a GAN was shown, but the present invention is not limited to this. In the present invention, as long as it is possible to restore a real vehicle color image from a real vehicle black-and-white image, it is also possible to create a coloring AI using deep learning or the like other than a GAN.

[0058] In the above embodiment, a system including an image processing apparatus and a vehicle was taken as an example, and an example in which an object detector is provided in the vehicle was described, but the present invention is not limited to this. In the present invention, the object detector may be provided in a device other than the vehicle.

[0059] All of the above embodiments are examples of the application of the present invention, and it goes without saying that any other embodiments within the scope described in the claims are also included in the technical scope of the invention.

Explanation of Signs

[0060] 10: Vehicle 11: ECU 12: Object detector 13: Network I / F 20: Image processing apparatus 21: CPU 22: Storage unit 23: Network I / F 24: Image processing unit 30a: Real vehicle color image (real image) 30b: Real vehicle black-and-white image (processed image) 31: Coloring AI (trained model) 32a: Simulation color image (first virtual image) 32b: Simulation black-and-white image (second virtual image) 33: Object detection training image (third virtual image) 241: Real vehicle black-and-white image creation unit (processed image creation unit) 242: Coloring AI creation unit (trained model creation unit) 243: Simulation color image creation unit (first virtual image creation unit) 244: Simulation black-and-white image conversion unit (second virtual image conversion unit) 245: Object detection training image creation unit (third virtual image conversion unit) 246: Annotation information extraction unit

Claims

A method for creating a dataset of an image processing apparatus having an image processing unit, comprising: Preparing a real image, which is a color image captured by an imaging device provided on a vehicle; Creating a processed image, which is a black-and-white image, by converting the real image into black-and-white by the image processing unit; Creating a learned model for restoring the real image from the processed image by the image processing unit using the real image and the processed image; Creating a first virtual image, which is a color image, by the image processing unit; Converting the first virtual image into a second virtual image, which is a black-and-white image corresponding to the processed image, by the image processing unit; Converting the second virtual image into a third virtual image by the image processing unit coloring the second virtual image using the learned model; Extracting annotation information of the third virtual image by the image processing unit. A method for creating a dataset of an image processing apparatus, characterized by the above. The method for creating a dataset of an image processing apparatus according to claim 1, wherein the third virtual image is a training image that becomes learning data for detecting an object from an image captured by the imaging device provided on the vehicle.

3. The learned model is created by a GAN (Generative Adversarial Networks) including a generator, a discriminator, and a prediction network. The method for creating a dataset of an image processing apparatus according to claim 1 or 2, characterized by the above. A processing image creation unit that creates a processed image, which is a black-and-white image, by converting a real image, which is a color image captured by an imaging device provided on a vehicle, into black-and-white; A learned model creation unit that creates a learned model for restoring the real image from the processed image using the real image and the processed image; A first virtual image creation unit that creates a first virtual image, which is a color image; A second virtual image conversion unit that converts the first virtual image into a second virtual image, which is a black-and-white image corresponding to the processed image; A third virtual image conversion unit that converts the second virtual image into a third virtual image by coloring the second virtual image using the learned model; An annotation information extraction unit that extracts annotation information of the third virtual image. An image processing apparatus, characterized by the above.

Images