IMAGE GENERATION METHOD, MACHINE LEARNING METHOD, IMAGE GENERATION DEVICE, PROGRAM, AND LEARNING IMAGE GENERATION METHOD

Abstract

According to the present invention, training images for building an image recognition model are easily prepared, and the necessary time period for data collection of images of defective products to be used as training images is reduced. This training image generating method includes: creating a SinGAN model including a generator and a discriminator for each of a plurality of layers on the basis of a first image in which a portion of interest is represented partially in a target object; generating an input image by combining a target object image and a portion-of-interest image; and generating a second image in which the form of the portion of interest is different from the first image, on the basis of the SinGAN model and the input image.

Description

[Technical field]

[0001] The present invention relates to an image generation method, a machine learning method, an image generation device, a program, and a method for generating images for learning. [Background technology]

[0002] With the recent development of deep learning technology, image recognition technology based on machine learning models is now being used for visual inspection of products manufactured in factories and other facilities. Summary of the Invention [Problem to be solved by the invention]

[0003] The above machine learning model may be applied to the detection of defective products, for example, but in order to improve the accuracy of the recognition of whether or not a product is defective, it is necessary to prepare a large number of images of defective products. However, in actual workplaces, the occurrence frequency of defective products is often low, making it difficult to secure a sufficient number of images of defective products.

[0004] In addition, since products such as food products are produced in different states (shape, type of ingredients, position, etc.) during production, product images in various states may be required to judge the finished product (for example, the quality of the presentation).

[0005] However, preparing a sufficient number of images of defective products and manufacturing a large number of products in various states is time-consuming and requires a long period of time.

[0006] The problem to be solved by the present invention is to easily prepare learning images for constructing an image recognition model and product images in various states, and to shorten the time required for collection. [Means for solving the problem]

[0007] In an image generation method according to one aspect of the present disclosure, a SinGAN model including a generator and a discriminator at each of multiple layers is created based on a first image in which a portion of an object is partially depicted, an input image is generated by synthesizing an image of the object and an image of the portion of interest, and a second image having a different appearance of the portion of interest from the first image is generated based on the SinGAN model and the input image.

[0008] In addition, in an image generation method according to another aspect of the present disclosure, when generating the output image, the input image is input to a generator of an intermediate layer of the multiple layers.

[0009] In addition, in an image generating method according to another aspect of the present disclosure, when generating the input image, the input image is generated by cutting out the region of the portion of interest and a periphery of the portion of interest from a synthesized image.

[0010] In addition, in an image generating method according to another aspect of the present disclosure, the intermediate layer generator is determined according to a layout of the portion of interest represented in the input image.

[0011] In addition, in an image generation method relating to another aspect of the present disclosure, when generating the input image, area information of the part of interest is obtained, and when generating the second image, the input image is input to the SinGAN model to generate an output image in which the aspect of the part of interest differs from that of the first image, and a second image including the part of interest included in the output image is generated based on the area information.

[0012] In addition, in an image generation method according to another aspect of the present disclosure, when generating the output image, random noise is input to at least the generator at the lowest hierarchical level, and an output image including the image of the area of ​​interest is output from the generator at the highest hierarchical level.

[0013] In an image generating method according to another aspect of the present disclosure, when generating the second image, random noise is input to at least a generator in a lowest hierarchy, and the second image is output from a generator in a highest hierarchy.

[0014] In an image generating method according to another aspect of the present disclosure, the portion of interest is a defective portion partially represented in an object.

[0015] In addition, in a machine learning method according to another aspect of the present disclosure, a machine learning model is trained based on the second image generated as described above, in which a photographed image of a product is input and a determination result is output as to whether the product is a good product or a defective product containing the defective part.

[0016] In addition, an image generating device relating to another aspect of the present disclosure includes a SinGAN model created based on a first image in which a portion of an object is depicted as a focus area, and including a generator and a discriminator in each of multiple layers, and an input image generation unit that generates an input image by synthesizing an object image and an image of the focus area, and generates a second image, the appearance of which differs from that of the first image, based on the SinGAN model and the input image.

[0017] In addition, a program relating to another aspect of the present disclosure causes a computer to operate as an image generation device that creates a SinGAN model including a generator and a discriminator at each of multiple layers based on a first image in which a portion of an object is shown, synthesizes the object image and the image of the portion of interest to generate an input image, and generates a second image, the appearance of the portion of interest differing from that of the first image, based on the SinGAN model and the input image.

[0018] In addition, in a training image generation method relating to another aspect of the present disclosure, a SinGAN model including a generator and a discriminator in each of multiple layers is created based on a first training image in which a defective part is partially shown in an object, and a second training image in which the state of the defective part is different from that of the first training image is generated using the SinGAN model. Effect of the Invention

[0019] According to the present disclosure, it is possible to easily prepare learning images for building an image recognition model and product images in various states, and also to shorten the time required for collection. [Brief description of the drawings]

[0020] [Figure 1] FIG. 1 is a diagram illustrating an example of a hardware configuration of a learning image generating device and a machine learning device. [Diagram 2] FIG. 1 is a functional block diagram showing an overall configuration of a determination system. [Diagram 3] FIG. 1 is a diagram illustrating the configuration of a SinGAN model. [Figure 4] FIG. 2 is a diagram for explaining image conversion according to the first embodiment. [Diagram 5] FIG. 2 is a diagram for explaining image conversion according to the first embodiment. [Figure 6] FIG. 1 is a flow diagram of a learning image generating method. [Figure 7] FIG. 1 is a flow diagram illustrating a method of random generation. [Figure 8] FIG. 11 is a flow diagram showing a method for generating an image including a defect. [Figure 9] FIG. 11 is a diagram showing a GUI according to a second embodiment. [Figure 10] FIG. 1 is a functional block diagram showing an overall configuration of an image generating system. [Figure 11] FIG. 13 is a diagram illustrating image conversion according to the third embodiment. [Figure 12] FIG. 13 is a diagram illustrating image conversion according to the third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0021] Hereinafter, preferred embodiments for carrying out the present invention (hereinafter, referred to as embodiments) will be described with reference to the drawings.

[0022] [First embodiment] Fig. 1 is a diagram showing an example of the hardware configuration of a learning image generating device 100 and a machine learning device 102. Fig. 1 shows a general computer, in which a processor such as a CPU (Central Processing Unit) 104, a memory such as a RAM (Random Access Memory) 106, an external storage device 108, a display device 110, an input device 112, and an I / O (Input / Output) 114 are connected to each other via a data bus 116 so as to exchange electrical signals. Note that the hardware configuration of the computer shown here is an example, and other configurations may also be used.

[0023] The external storage device 108 is a device capable of statically recording information, such as a hard disk drive (HDD) or a solid state drive (SSD). The display device 110 is a cathode ray tube (CRT) or a so-called flat panel display, and displays images. The input device 112 is one or more devices, such as a keyboard, a mouse, or a touch panel, that allow a user to input information. The I / O 114 is one or more interfaces that allow the computer to exchange information with external devices. The I / O 114 may include various ports for wired connection and a controller for wireless connection.

[0024] A program for causing a computer to function as the learning image generating device 100, the machine learning device 102, and the image generating device 1002 (see FIG. 10) is stored in an external storage device 108, and is read into the RAM 106 as necessary and executed by the CPU 104. That is, the RAM 106 stores codes for implementing various functions shown as functional blocks in FIG. 1 by being executed by the CPU 104. The program may be provided by being recorded on an information recording medium that can be read by a computer, such as an optical disk, a magneto-optical disk, or a flash memory, or may be provided via an external information communication line, such as the Internet, via the I / O 114.

[0025] FIG. 2 is a functional block diagram showing the overall configuration of the determination system 200 of the present disclosure. As shown in FIG. 2, the determination system 200 includes an image database 202, an input image generation unit 204, a first learning unit 206, a SinGAN model unit 208, a database for determination device 210, and a determination device 212. The learning image generation device 100 includes the input image generation unit 204, the first learning unit 206, and the SinGAN model unit 208. The machine learning device 102 includes the first learning unit 206 and the SinGAN model unit 208. Here, the "machine learning device 102" refers to a device that executes learning of the SinGAN model unit 208 using the first learning image, and the "learning image generation device 100" refers to a device that generates a second learning image, which is teacher data used for learning in the determination model 226 in which supervised learning is performed.

[0026] The SinGAN model unit 208 includes one or more SinGAN models (see Shaham, Tamer Lott, Tali Dickel, and Tomer Michaelis, Singan: Learning a Generative Model from a Single Natural Image, Proceedings of the International Conference on Computer Vision 2019). In FIG. 2, the SinGAN model unit 208 includes n SinGAN models from the first SinGAN model 222A to the n-th SinGAN model 222N. In the following, the first SinGAN model 222A and the second SinGAN model 222B are mainly included.

[0027] Furthermore, the "first learning image" is one form of the first image described later (see the third embodiment) and is training data used for training the SinGAN model according to this embodiment. The "second learning image" is one form of the second image described later (see the third embodiment) and is training data used for training an image recognition model (e.g., a machine learning model implemented in an inspection device used for visual inspection). Furthermore, as an example of the first learning image and the second learning image, a case will be described in which an image showing a partially defective part in an object is used.

[0028] Moreover, the target object is, for example, a product to be inspected in an inspection process in a factory, such as a casting formed by pouring into a mold. A defective part is one form of a part of interest described later (see the third embodiment), such as a scratch, defect, or cavity (a cavity formed by a change in volume when solidifying from a molten state) that appears on the surface of the product. As an example of the determination device 212, a device that determines whether or not a scratch, defect, or cavity is formed in a casting formed by pouring into a mold in an inspection process in a factory will be used for explanation.

[0029] Moreover, an image input to the SinGAN model is referred to as an "input image," and an image output from the SinGAN model is referred to as an "output image." Furthermore, an image showing a defective part is generally referred to as a "defective image," and an image in which only the defective part is extracted from the defective image is referred to as a "defective part image." Furthermore, an image showing an object (good product) with no defective parts is referred to as a "good product image." Furthermore, an image showing an object, regardless of whether it is a good product or a defective product, is generally referred to as an "object image."

[0030] The image database 202 is a database that stores a plurality of captured images of good products and at least one captured image of a defective product. Specifically, for example, the image database 202 stores images of good products and images of defective products captured in an appearance inspection process. In a product manufacturing process, usually, more good products are manufactured than defective products. Therefore, the image database 202 stores more images of good products than defective images. The image database 202 stores defective images equal to or greater than the number of SinGAN models to be trained. The image database 202 may be, for example, a storage device included in a computer that manages the entire manufacturing process, or may be a storage device included in a computer that can communicate with an imaging unit that captures good products and defective images via a network.

[0031] The first learning unit 206 included in the machine learning device 102 learns a SinGAN model using the first training image. The training image generating device 100 uses the trained SinGAN model to generate second training images having a different manner of defect parts from the first training images. When the SinGAN model unit 208 includes multiple SinGAN models, the first learning unit 206 learns each SinGAN model using different first training images. In this case, the training image generating device 100 uses each trained SinGAN model to generate second training images having a different manner of defect parts from the first training images used for training.

[0032] The SinGAN model is a machine learning model that receives an input image and generates a second learning image having a different defective part pattern from the first learning image used for learning. Specifically, FIG. 3 is a diagram showing the configuration of one SinGAN model. Note that the SinGAN model itself is known, so the explanation will be kept to a minimum.

[0033] As shown in FIG. 3, the SinGAN model is composed of multiple layers, each of which includes a GAN (Generative Adversarial Network). The number of layers may be any number equal to or greater than two. A GAN includes two neural networks called a generator and a discriminator. The generator included in the top layer is generator G 0 , the discriminator is discriminator D 0 The generator in the i-th layer is the generator G i , the discriminator is discriminator D i The generator at the lowest level is generator G N , the discriminator is discriminator D N Each layer except the top layer also includes an upsampler 302 that increases the resolution of the image output by the generator.

[0034] The generators included in each layer generate an image in which a defective part is partially shown in the object and which is difficult to distinguish from the first learning image. N accepts input of random noise of a given size. Then, the generator G N The discriminator D outputs an image that partially shows a defective part in the object and has the same size as the random noise. N The generator G N An image of the object generated by the generator G N The upsampler 302 receives both the random noise input to the generator G and a first learning image (correct image) downsampled (reduced in resolution) to the same size as the random noise input to the generator G. N Improves the resolution of the image output by Generator G, which is one level higher. N-1 Output to.

[0035] The generators of each layer except the lowest layer accept input of an image output by the upsampler 302 of the layer one level below and random noise of the same resolution as the image. The generators of each layer then output an image of the same size as the input image. Meanwhile, the discriminators of each layer receive input of both an image (incorrect image) of an object with a partially defective part generated by the generator of the same layer, and a first learning image (correct image) downsampled (reduced resolution) to the same size as the image.

[0036] The output of the discriminator in each layer is used to determine whether the input data is an incorrect image or a correct image. The first learning unit 206 performs learning from the lowest layer to the highest layer so that the discriminator included in each layer can correctly discriminate between the two, and the generator so that the discriminator cannot discriminate between the two.

[0037] The SinGAN model has the above configuration, and can perform learning using one first learning image. In addition, in the SinGAN model, the lower the layer, the smaller the image is input, and the higher the layer, the larger the image is input. The SinGAN model has a feature that the lower the layer, the closer the image to the training image is generated, whereas the higher the layer, the more the image is generated with the input image unchanged in appearance. Therefore, by inputting an input image to a generator of an appropriately selected layer, which will be described later, it is possible to generate a style-converted image in which the hue, etc., is converted while maintaining the defective area represented in the input image.

[0038] The input image generating unit 204 includes a random noise generating unit 214, an extracting unit 216, a cropping unit 218, and a synthesizing unit 220. The random noise generating unit 214 generates random noise with a predetermined resolution. Here, the predetermined resolution is a resolution at which the generators of each layer included in the SinGAN model can be accepted. The random noise generating unit 214 appropriately generates random noise with a resolution according to the layer of the generator.

[0039] The extraction unit 216 receives a defective image and extracts a defective part image that is an image of only the defective part shown in the defective image. Specifically, for example, as shown in FIG. 4(a), the extraction unit 216 receives defective images in which defective parts are partially shown in various objects. The defective image may be an image of an object in which a defect is actually photographed, or an image generated by a trained SinGAN model in which random noise is input. Then, the extraction unit 216 extracts only an area in which a defective part is shown as the defective part image from the defective image. The number of defective images input to the extraction unit 216 and the number of defective part images extracted by the extraction unit 216 are arbitrary.

[0040] The cropping unit 218 receives an image in which a defective part is partially shown in the target object, and crops out the defective part and the periphery of the defective part. Specifically, for example, as shown in FIG. 4(b), when the cropping unit 218 receives a photographed defective image or a defective image synthesized by the synthesis unit 220, the cropping unit 218 crops out a certain area including the defective part shown in the defective image. Here, the cropping unit 218 crops out an image of a size suitable for the generator of the highest layer of the trained SinGAN model. Note that the cropping size does not have to match the image size accepted by the generator of the highest layer. For example, it is sufficient that the size is close to the image size accepted by the generator of the highest layer (for example, the enlargement / reduction ratio is within 10%). When the input image includes multiple defective parts, the cropping unit 218 may crop out the image so that all the defective parts are included, or may crop out the image so that only some of the defective parts are included. The image cropped by the cropping unit 218 is input to the generator of the middle layer of the SinGAN model after being resized to a size suitable for the generator. Furthermore, the cropping unit 218 may obtain area information that indicates the area of ​​the cropped image in the image before cropping (hereinafter also referred to as the entire image).

[0041] The synthesis unit 220 synthesizes the object image and the defective part image. Specifically, for example, as shown in FIG. 4(c), the synthesis unit 220 synthesizes the good product image and the defective part image by overwriting a partial area of ​​the good product image with the defective part image extracted by the extraction unit 216. The good product image is an image of various objects that do not have defects, or an image of an object that does not include defects, generated by a known means. When synthesizing, the synthesis unit 220 may change the size, aspect ratio, hue, and angle of the defective part image and then synthesize it with the good product image. In addition, the area to which the defective part image is pasted may be any area as long as it is an area in which the object of the good product image is represented.

[0042] The synthesis unit 220 may also acquire area information of the defective part. Specifically, when synthesizing the non-defective product image and the defective part image, the synthesis unit 220 may acquire area information representing an area of ​​the defective part image that is occupied in the synthesized image. When the size, aspect ratio, and angle of the defective part image are changed during synthesis, the area information is information representing an area in the synthesized image where the changed defective part image is pasted.

[0043] When the region information is acquired, an image including a defective part image included in the output image output from the generator may be generated based on the region information. Specifically, for example, as shown in FIG. 4(b), the cropping unit 218 first crops an input image from the whole image and acquires region information. Next, as shown in FIG. 5(a), the SinGAN models 222A and 222B input the cropped image to the generator and output an image (output image) that has been subjected to style conversion processing.

[0044] 5(b), the synthesis unit 220 may synthesize the region other than the defective part of the input image with the defective part of the output image based on the region information. This makes it possible to generate an image in which only the defective part region is replaced with an image generated by the SinGAN model from the image around the defective part.

[0045] Furthermore, the synthesis unit 220 may synthesize the image before being cut out by the cutout unit 218 and the image synthesized by the synthesis unit 220 based on the region information. Specifically, as described above, the region information is information representing the region of the cut out image in the image before being cut out by the cutout unit 218. As shown in FIG. 5(c), the synthesis unit 220 synthesizes the image synthesized by the synthesis unit 220 with the region represented by the region information of the image before being cut out by the cutout unit 218. This makes it possible to generate an image in which only the defective region is replaced with the image generated by the SinGAN model.

[0046] The determination device database 210 is a database that stores a plurality of non-defective images, at least one defective image, and a second learning image. Specifically, for example, the determination device database 210 stores all of the non-defective and defective images stored in the image database 202 and the second learning image generated by the learning image generating device 100. The determination device database 210 may be, for example, a storage device included in a computer that manages the entire manufacturing process, or may be a storage device included in a computer that can communicate with the determination device 212 via a network.

[0047] The determination device 212 includes a second learning unit 224 and a determination model 226. Specifically, for example, the determination device 212 is a device that determines whether or not a flaw, a defect, or a void is formed in a casting formed by pouring into a mold in an inspection process in a factory.

[0048] A second learning unit 224 included in the determination device 212 learns a determination model 226 using images stored in the determination device database 210. The determination model 226 is a machine learning model that receives an image of a product taken in an inspection process and outputs a determination result as to whether the product is a good product or a defective product. The determination model 226 may be a known machine learning model, such as a CNN (Convolutional Neural Network). When a product image taken in an inspection process is input, the trained determination model 226 outputs a determination result as to whether or not a defective part is shown in the product image.

[0049] 6 and 7 are flow diagrams of a training image generating method by the training image generating device 100 according to the present embodiment. First, a first training image used to create a SinGAN model is acquired (S602). Specifically, the first learning unit 206 acquires at least one defective image in which a defective part is partially shown in various objects from the image database 202. As described above, the defective image is preferably an image that is actually taken of an object in which a defect is formed, but if a trained SinGAN model exists, the defective image may be an image generated by inputting random noise into the trained SinGAN model. In addition, a plurality of first training images may be acquired, and here, it is assumed that two first training images are acquired.

[0050] Next, the first learning unit 206 executes learning of the SinGAN model (S604). The first learning unit 206 executes learning of the generator and discriminator of each layer in order from the lowest layer to the highest layer by the above learning method using the two first learning images acquired in S602. In the above, since two first learning images are acquired, the first learning unit 206 executes learning of the first SinGAN model 222A using the first first learning image, and executes learning of the second SinGAN model 222B using the second first learning image. Here, since the first first learning image and the second first learning image are different, the first SinGAN model 222A and the second SinGAN model 222B are different.

[0051] Next, if random generation is not performed, proceed to S607, and if random generation is performed, proceed to S608 (S606). Here, random generation refers to inputting random noise into a trained SinGAN model to randomly generate defective images. The user may appropriately select whether or not to perform random generation.

[0052] In S607, the input image generating unit 204 acquires one or more defect images in which defective parts are partially shown in various objects from the image database 202. The defect images are images obtained by actually photographing objects in which defects are formed.

[0053] Meanwhile, in S608, random generation is performed. FIG. 7 is a flow diagram showing a method of random generation. Specifically, first, the random noise generation unit 214 generates random noise as shown in FIG. 5(d) (S702). The generated random noise is generated with an image size suitable for at least the generator of the lowest layer of the SinGAN model. In FIG. 5(d), three times as many random noises as the layers of the first SinGAN model 222A are generated for the first SinGAN model 222A, and three times as many different random noises as the layers of the second SinGAN model 222B are generated for the second SinGAN model 222B. However, the number of random noises to be generated may be appropriately set according to the number of second learning images required.

[0054] Next, the generated random noise is input to the trained SinGAN model (S704). Specifically, for example, as shown in FIG. 5(d), the generated random noise for the first SinGAN model 222A is input to the generator G of all layers of the first SinGAN model 222A trained in S604. 0 From G N The first SinGAN model 222A inputs three defective images to the top-level generator G 0 Similarly, the remaining random noise for the first SinGAN model 222B is generated by the generator G 0 From G N The second SinGAN model 222B inputs three defective images to the top-level generator G 0 Output from.

[0055] Here, when different random noises are input to the same SinGAN model, images are output that are similar to the first learning image used for learning, although the number and size of the defects are different. Also, the first SinGAN model 222A and the second SinGAN model 222B are trained using different first learning images. Therefore, as shown in the upper and lower rows of the output image in FIG. 5(d), the three defective images generated by the first SinGAN model 222A and the three defective images generated by the second SinGAN model 222B have different styles, such as the hue and size of the object.

[0056] In addition, when random generation is used, in addition to the defective image generated using the SinGAN model, a defective image obtained from the image database 202 may be obtained before or after S608. In this case, the second learning image is generated based on both the captured defective image and the generated defective image.

[0057] After the defective image is acquired or generated in S607 or S608, the process proceeds to S610 if the defective part image is to be extracted, and proceeds to S612 if the defective part image is not to be extracted (S609). In S610, the extraction unit 216 extracts the defective part image, and then generates an image including the defective part. FIG. 8 is a flow diagram showing a method for extracting a defective part image. The extraction unit 216 receives a defective image and extracts a defective part image that is an image of only the defective part shown in the defective image (S802). Specifically, for example, as shown in FIG. 4(a), the extraction unit 216 receives the defective image acquired in S607 or the defective image generated in S608, and extracts only the area showing the defective part as the defective part image. Note that in FIG. 4(a), only defective part images showing some of the defective parts are extracted from among the defective parts included in the six defective images, but the extraction unit 216 may extract defective part images showing all of the defective parts included in the input defective image.

[0058] Next, the synthesis unit 220 changes the size, angle, and hue of the extracted defective part image (S804). The size, angle, and hue are appropriately changed using a known technique.

[0059] Next, the synthesis unit 220 synthesizes the good product image and the defective part image (S806). Specifically, for example, as shown in Fig. 4(c), the synthesis unit 220 acquires a good product image from the image database 202. The synthesis unit 220 then synthesizes the good product image and the defective part image by overwriting a partial area of ​​the acquired good product image with the defective part image extracted by the extraction unit 216. At this time, the synthesis unit 220 acquires area information indicating the area of ​​the defective part image that is occupied by the synthesized image.

[0060] Next, if the image is to be cropped, the process proceeds to S614, and if the image is not to be cropped, the process proceeds to S622 (S612). Here, cropping the image refers to a process performed by cropping unit 218 to crop the area of ​​the defective part and the periphery of the defective part. S612 to S624 are repeatedly executed as many times as the number of defective images acquired or generated in S607 to S610. For example, if six defective images are acquired in S608, S612 to S624 are executed six times.

[0061] The user may appropriately select whether or not to crop the image. For example, the selection may be made based on the proportion of the defective part that occupies the defective image. Specifically, the image may not be cropped if the size of the defective part is 30% or more of the entire defective image, and may be cropped if the size is less than 30%. In the style conversion process of S616, if the proportion of the defective part that occupies the defective image is too small, the difference between the images before and after the style conversion process becomes small. By selecting whether or not to crop based on the proportion, a defective image that shows a defective part in a form not present in the image database 202 can be generated by the style conversion process of S616.

[0062] When the defective image is input to the cropping unit 218, as shown in FIG. 4(b), the cropping unit 218 crops out a certain area including the defective part shown in the defective image (S614). Here, the cropping unit 218 crops out the image to a size suitable for the generator of the highest layer of the trained SinGAN model. The cropping size does not have to match the image size accepted by the generator of the highest layer. The cropped image is an input image to the SinGAN model. The cropping unit 218 also acquires area information indicating the area of ​​the cropped image in the image before cropping. Then, the cropping unit 218 resizes the cropped image to a size suitable for the generator of the predetermined intermediate layer of the multiple layers and inputs it.

[0063] Next, the SinGAN model generates an output image in which the style conversion process is performed on the input image (S616). Specifically, for example, the SinGAN model includes a generator G 0 Then, an output image is generated by performing style conversion processing on the input image.

[0064] Here, when the input image is an image generated using a SinGAN model or a part of the image, the input image is input to a SinGAN model different from the SinGAN model used to generate the input image. Specifically, for example, when the input image is a defective image generated using the first SinGAN model 222A in S608 or an image cut out from the defective image, the input image is input to the second SinGAN model 222B. When the input image is a defective image generated using the second SinGAN model 222B in S608 or an image cut out from the defective image, the input image is input to the first SinGAN model 222A. By using a SinGAN model different from the SinGAN model generating the input image, an output image in which the style of the input image is changed can be generated.

[0065] Next, as shown in FIG. 5(c), the synthesis unit 220 synthesizes the region other than the defective part of the input image with the defective part of the output image based on the region information (S618). Here, the region in which the defective part is expressed in the input image is represented by the region information acquired in S806 and S614. When generating the synthesis image in S618, the region in which the defective part is expressed in the input image needs to match the region in which the defective part is expressed in the output image. Therefore, in S616, a generator of an intermediate layer that performs style conversion to an extent that the region in which the defective part is expressed in the input image does not change is selected. That is, the layer of the generator to which the cut-out image is input is determined according to the layout of the defective part expressed in the input image. In other words, by determining the intermediate layer to be input according to the layout of the defective part, style conversion can be performed to an extent that the region in which the defective part is expressed in the input image does not change.

[0066] Next, based on the area information acquired in S614, the composition unit 220 composes the image before cropping by the cropping unit 218 in S614 with the image composited by the composition unit 220 in S618 to generate a second learning image (S620). At this time, if the size of the image cropped in S614 differs from the size of the image composited by the composition unit 220 in S618, the composition unit 220 may enlarge or reduce the composite image based on the area information acquired in S614.

[0067] In S612, if the image is not cropped, the defective image acquired in S607 or the defective image generated in S608 or S610 is input to the SinGAN model. The SinGAN model is then fed to the generator G 0 An output image in which style conversion processing has been performed on the input image is generated as a second learning image (S622). Even if the image is not cropped, it may be resized to match the input size of the intermediate layer generator and input. Also, as in S616, the SinGAN model used in S622 is a SinGAN model different from the SinGAN model used to generate the input image.

[0068] As is clear from the example of the casting cavity shown in this example, it is not easy to actually prepare a sufficient number of appropriate learning images for training a neural network model. In order to obtain learning images showing various different defect modes, it is necessary to realize the various different defect modes, but this is not realistic because it requires too much time and cost. However, according to this embodiment, it is possible to easily create a large number of learning images showing different defect modes. For example, the time required for starting up a new appearance inspection process is shortened.

[0069] 2 illustrates a configuration in which the machine learning device 102 is part of the training image generating device 100, but the machine learning device 102 may be prepared as a physically independent device. The training image generating device 100 and the machine learning device 102 may be incorporated as part of another machine or device, or may be appropriately configured using the physical configuration of another machine or device as necessary. More specifically, the training image generating device 100 and the machine learning device 102 may be implemented by software using a general computer.

[0070] Furthermore, the programs that operate a computer as the training image generating device 100 and the machine learning device 102 may be integrated, or each may be executed independently. Furthermore, the programs may be incorporated as modules into other software. Furthermore, the training image generating device 100 and the machine learning device 102 may be constructed on a so-called server computer, and only their functions may be provided to a remote location via a public telecommunications line such as the Internet.

[0071] [Second embodiment] Next, an image generating method according to the second embodiment will be described. The image generating method according to the second embodiment is a method for generating an image for learning, similar to the first embodiment. In the second embodiment, instead of steps S606 to S610 in the first embodiment, a user specifies the shape of the defective part, thereby generating an image including a defective part of any shape. The other steps of the image generating method according to the second embodiment are similar to those in the first embodiment.

[0072] First, a SinGAN model is created and trained. This step is similar to S602 and S604 included in the first embodiment.

[0073] Next, a defective part image having a shape specified by the user is generated. Specifically, a description will be given with reference to Fig. 9(a) to Fig. 9(c). Fig. 9(a) to Fig. 9(c) are an example of a GUI (Graphical User Interface) displayed on the display device 110 when a user instructs a computer to execute a program for generating a defective part image of an arbitrary shape. As shown in Fig. 9(a) to Fig. 9(c), the GUI displays a read button 902, a save button 904, a color designation drop-down button 906, an eyedropper button 908, a generate button 910, a defective image field 912, a designated shape defective image field 914, and a shape designation field 916.

[0074] First, the user specifies a defective image. Specifically, for example, the user clicks the load button 902 and specifies a defective image stored in the image database 202. The specified defective image is displayed in the defective image field 912 and the shape specification field 916 as shown in FIG. 9(a). Note that the specified image is the defective image used in the training of the SinGAN model, but is not limited to this and may be another defective image.

[0075] Next, the user specifies a color. Specifically, for example, the user operates the color specification drop-down button 906 and specifies a color by selecting a color displayed in a list. Here, the specified color is a color close to the color of the defective part included in the image used for learning the SinGAN model 222. The color close to the color of the defective part is, for example, a color whose distance on the chromaticity coordinate from the color of the defective part is equal to or less than a predetermined value. In addition, the user may operate the eyedropper button 908 to select any part of the defective part shown in the defective image in the shape specification field 916. The color of the defective image at the selected part becomes the color specified by the user. Note that the color is expressed by, for example, each gradation of red, blue, and green, and the user may specify the color by inputting the numerical value of each gradation.

[0076] Next, the user specifies a shape. Specifically, for example, the user specifies a desired shape by dragging the mouse in the shape specification field 916. A dashed line 918 shown in the shape specification field 916 of Fig. 9(b) is an example of a shape specified by the user.

[0077] Next, the user generates a defective part image. Specifically, the user clicks the generate button 910. When the generate button 910 is clicked, as shown in FIG. 9(c), the area surrounded by the dashed line 918 in the shape designation field 916 is given the designated color, and the image with the designated shape and the designated color is displayed in the designated shape defective image field 914. Furthermore, when the user clicks the save button 904, the image displayed in the designated shape defective image field 914 is saved as a defective part image. The saved defective part image corresponds to the defective part image extracted by the extraction unit 216 in S802 included in S610 of the first embodiment.

[0078] Then, the synthesis unit 220 rotates the generated defective part image and synthesizes the good product image and the rotated defective part image. This step is the same as S804 and S806 in the first embodiment. Through the above steps, a defective image is generated in which an image having an arbitrary shape desired by the user and a color close to the color of the defective part included in the image used for learning is synthesized with the good product image.

[0079] The subsequent steps are the same as steps S612 to S624 in the first embodiment, and therefore the description thereof will be omitted. As described above, since the shape and color can be specified while referring to the defective part shown in the defect image, the user can generate a defective part image showing a defective part having a shape that does not actually occur.

[0080] [Third embodiment] Next, an image generating system 1000 and an image generating method according to the third embodiment will be described. Unlike the first embodiment, the image generating system 1000 and the image generating method according to the third embodiment are an apparatus and a method for generating an image that is not limited to learning applications and can be applied to other applications. Note that a description of the same points as the first embodiment will be omitted.

[0081] FIG. 10 is a functional block diagram showing the overall configuration of an image generating system 1000 of the present disclosure. As shown in FIG. 10, the image generating system 1000 includes an image database 202, an input image generating unit 204, a first learning unit 206, a SinGAN model unit 208, and a second image database 1004. The image generating device 1002 includes the input image generating unit 204, the first learning unit 206, and the SinGAN model unit 208. The machine learning device 102 includes the first learning unit 206 and the SinGAN model unit 208. Here, the "machine learning device 102" refers to a device that executes learning of the SinGAN model unit 208 using a first image, and the "image generating device 1002" refers to a device that generates a second image.

[0082] Moreover, the "first image" is training data used for learning the SinGAN model according to this embodiment. The "second image" is an image generated by the image generating device 1002, and the generated image itself is used as is or after processing. Moreover, as an example of the first image and the second image, a case will be described in which an image in which a part of an object is shown as a focus area is used.

[0083] The target object is, for example, a product manufactured in a factory or a store, such as food products such as pizza, mashed potatoes, salad, cake, etc., or industrial products such as tableware, etc. As an example of the second image, an image displayed on a menu presented in a restaurant, in which pizza with various toppings is shown, will be used for explanation.

[0084] Moreover, an image showing a portion of interest is generally referred to as an "image of interest," and an image in which only the portion of interest is extracted from the image of interest is referred to as an "image of portion of interest." An image showing an object without a portion of interest is referred to as an "image not including a portion of interest." Moreover, an image showing an object, whether or not it includes a portion of interest, is generally referred to as an "image of object."

[0085] Image database 202 is a database that stores a plurality of captured images that do not include a portion of interest and at least one captured image of interest. Specifically, for example, image database 202 stores images of only pizza dough captured during the manufacturing process, and images of whole or piece pizzas with toppings on them that have been baked.

[0086] The first learning unit and the SinGAN model unit 208 are the same as those in the first embodiment. Note that, as the first image used for learning, an image of a baked pizza with toppings on it is used, rather than an image of only the dough.

[0087] The input image generating unit 204 includes a random noise generating unit 214, an extracting unit 216, a cropping unit 218, and a synthesizing unit 220. The functions of each unit are the same as those in the first embodiment, except that the extracting unit 216 extracts an image of a portion of interest instead of an image of a defective portion, and the extracting unit 216 and the cropping unit 218 crop the image of interest instead of an image of a defective portion.

[0088] The second image database 1004 is a database that stores the generated second images. Note that the second image database 1004 may be omitted, and the image database 202 may store the generated second images.

[0089] In the third embodiment, the generated image does not need to be used for learning. Therefore, the image generating system 1000 according to the third embodiment does not include the determination device 212.

[0090] 11(a) to 12(b), an image generating method by an image generating device 1002 according to the third embodiment will be described. In this embodiment, a case will be described in which an input image is generated by cutting out a region of a portion of interest and its surroundings from a composite image.

[0091] First, a SinGAN model including a generator and a discriminator in each of multiple layers is created based on a first image in which a part of the target object is shown. This step is the same as S602 and S604 included in the first embodiment, except that learning is performed using the first image. As described above, an image of a baked pizza with toppings on it is used as the first image used for learning, rather than an image of only the dough. Here, the SinGAN model trained using an image of a pizza with only paprika on it as the first image is referred to as the first SinGAN model 222A. Also, the SinGAN model trained using an image of a pizza with only salami on it as the first image is referred to as the second SinGAN model 222B.

[0092] Next, an attention portion image having a shape specified by the user is generated. Specifically, as in the second embodiment, an image is generated that has a shape and color desired by the user and resembles the ingredients to be placed on pizza. The bottom left of FIG. 11(a) shows, as the attention portion images generated, a green image shaped like the number 3, which represents paprika, and a light red oval image, which represents salami. The images are attention portion images created by the user using the GUIs shown in FIG. 9(a) to FIG. 9(c).

[0093] Next, the synthesis unit 220 synthesizes the object image and the image of the part of interest to generate an input image. Specifically, the synthesis unit 220 acquires an image that does not include a part of interest from the image database 202. The synthesis unit 220 then synthesizes the image that does not include a part of interest and the image of the part of interest by overwriting a partial area of ​​the acquired image that does not include a part of interest with the image of the part of interest created by the user using a GUI. As shown in FIG. 11(a), the synthesis unit 220 synthesizes an image of paprika and an image of salami with an image of only dough. At this time, the synthesis unit 220 acquires area information that indicates the area of ​​the image of the part of interest that is occupied in the synthesized image.

[0094] Next, the cutout unit 218 cuts out a certain area including the attention part shown in the attention image. This step is the same as S614 in the first embodiment. As a result, as shown in FIG. 11(b), an attention image is generated in which a background is added to the image of paprika and the image of salami created by the user.

[0095] Next, an input image is input to the SinGAN model to generate an output image in which the aspect of the attention part is different from that of the first image. That is, the SinGAN model generates an output image in which the input image has been subjected to style conversion processing. This step is the same as S616 in the first embodiment. Specifically, as shown in FIG. 11(c), the first SinGAN model A to which an image of paprika is input generates an output image including paprika that has been subjected to style conversion processing. Also, the second SinGAN model B to which an image of salami is input generates an output image including salami that has been subjected to style conversion processing. This makes it possible to generate an attention image that reflects the shape created by the user and resembles the toppings of an actual baked pizza.

[0096] Next, the synthesis unit 220 generates a second image including the attention portion included in the output image based on the region information. That is, the synthesis unit 220 synthesizes the region other than the attention portion of the input image with the region of the attention portion of the output image based on the region information. This step is the same as S618 in the first embodiment. Specifically, as shown in FIG. 12(a), an image is generated in which the image of the ingredients after the style transfer process is synthesized with the background before the style transfer process is performed.

[0097] Next, the synthesis unit 220 generates a second image. Specifically, as shown in Fig. 12(b), based on the region information, the synthesis unit 220 synthesizes the image before the image is cut out by the cutout unit 218 (the image on the left side in Fig. 11(b)) and the image synthesized by the synthesis unit 220 (the image on the right side in Fig. 12(a)) to generate the second image.

[0098] Through the above steps, a second image having a different aspect of the part of interest from the first image can be generated based on the SinGAN model and the input image. In the first and second embodiments, a machine learning method was described in which the part of interest is a defective part partially represented on the object, and a photographed image of a product is input based on the second image (here, the second learning image), and a determination result is output as to whether the product is a good product or a defective product including a defective part. However, as in the third embodiment, the generated second image is not limited to learning purposes and may be used for other purposes (for example, an image to be written on a menu table). The third embodiment is particularly useful in the case of creating a large number of images having similar parts of interest but with a wide variety of shapes, numbers, arrangement positions, etc.

[0099] In the above, the case where the image of interest is created using the GUI shown in the second embodiment has been described, but in the third embodiment, the image of interest may be created using the same method as in the first embodiment. For example, in the third embodiment, the image of interest may be generated using random generation. In this case, a plurality of SinGAN models 222 are created using at least two different images on which the same type of ingredients are placed.

[0100] Specifically, for example, a first SinGAN model 222A is created by learning using an image of a pizza topped with only paprika as the first image. A second SinGAN model 222B is created by learning using an image of a pizza topped with only paprika, which is different in color and shape from the first image, as the first image. Then, a target image may be generated by inputting random noise to each of the first SinGAN model 222A and the second SinGAN model 222B.

[0101] Furthermore, in the third embodiment, when random generation is used, as in the first embodiment, during style transfer processing, the input image is input to a SinGAN model 222 different from the SinGAN model 222 used to generate the input image. That is, an input image generated based on an image of interest generated by the first SinGAN model 222A is input to the second SinGAN model 222B. On the other hand, an input image generated based on an image of interest generated by the second SinGAN model 222B is input to the first SinGAN model 222A. This allows the generation of a second image showing at least two peppers with different colors and shapes.

[0102] Additionally, in the third embodiment, similarly to the first embodiment, extraction of an image of a portion of interest (corresponding to 610) and generation and synthesis of cut-out images (S614 to S620) may be performed or omitted as appropriate. [Explanation of symbols]

[0103] 100 Learning image generation device, 102 Machine learning device, 104 CPU, 106 RAM, 108 External storage device, 110 Display device, 112 Input device, 114 I / O, 116 Data bus, 200 Judgment system, 202 Image database, 204 Input image generation unit, 206 First learning unit, 208 SinGAN model unit, 210 Judgment device database, 212 Judgment device, 214 Random noise generation unit, 216 Extraction unit, 218 Cutting unit, 220 Synthesis unit, 222A First SinGAN model, 222B Second SinGAN model, 222N n-th SinGAN model, 224 Second learning unit, 226 Judgment model, 302 Upsampler, 902 Load button, 904 Save button, 906 Color designation drop-down button, 908 Eyedropper button, 910 Generate button, 912, defective image field, 914, specified shape defective image field, 916, shape specification field, 918, dashed line, 1000, image generation system, 1002, image generation device, 1004, second image database.

Claims

1. A SinGAN model including a generator and a discriminator in each of a plurality of layers is created based on a first image in which a portion of interest is partially depicted in the object; The object image and the image of the target area are combined to generate an input image; generating a second image having a different aspect of the target portion from that of the first image based on the SinGAN model and the input image; Image generation method.

2. The image generation method according to claim 1 , wherein when generating an output image output from the SinGAN model, the input image is input to a generator of an intermediate layer of the plurality of layers.

3. The image generating method according to claim 2 , wherein the input image is generated by cutting out the region of the target area and a periphery of the target area from the combined image.

4. 4. The image generating method according to claim 2, wherein the intermediate layer generator is determined in accordance with a layout of the target portion represented in the input image.

5. When generating the input image, region information of the attention portion is acquired; When generating the second image, inputting the input image into the SinGA model to generate an output image having a different aspect of the target portion from that of the first image; generating a second image including the portion of interest included in the output image based on the region information; The image generating method according to any one of claims 1 to 4.

6. When generating an output image output from the SinGA model, Random noise is fed to at least the lowest level generator, outputting the output image including the target portion image from the generator at the highest level; The image generating method according to any one of claims 1 to 5.

7. When generating the second image, Random noise is fed to at least the lowest level generator, outputting the second image from the highest hierarchical generator; The image generating method according to any one of claims 1 to 4.

8. The image generating method according to claim 1 , wherein the target portion is a defect portion partially represented in the object.

9. A machine learning method in which a photographed image of a product is input based on the second image generated by claim 8, and a machine learning model is trained to output a determination result as to whether the product is a good product or a defective product containing the defective part.

10. a SinGAN model including a generator and a discriminator in each of a plurality of layers, the SinGAN model being created based on a first image in which a portion of interest is partially depicted in the object; an input image generation unit that generates an input image by combining the object image and the attention portion image, An image generating device that generates a second image, in which the aspect of the target portion is different from that of the first image, based on the SinGAN model and the input image.

11. Computer, A SinGAN model including a generator and a discriminator in each of a plurality of layers is created based on a first image in which a portion of interest is partially depicted in the object; The object image and the image of the target area are combined to generate an input image; generating a second image having a different aspect of the target portion from that of the first image based on the SinGAN model and the input image; A program that operates as an image generating device.

12. creating a SinGAN model including a generator and a discriminator in each of a plurality of layers based on a first learning image in which a defective portion is partially represented in the object; The target image and the defect image are combined to generate an input image; generating a second learning image having a different defect state from that of the first learning image based on the SinGAN model and the input image; A method for generating images for training.

Images