Data generation device, data generation method, and data generation program

The data generation device automates the creation of training images and labels for classifiers, addressing the challenge of preparing large numbers of defective part samples by synthesizing and labeling images, thereby enhancing the efficiency and accuracy of defective part detection.

JP7576772B2Active Publication Date: 2024-11-01PANASONIC INTELLECTUAL PROPERTY MANAGEMENT CO LTD
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
JP2020213641
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Filing Date
2020-12-23
Publication Date
2024-11-01
Estimated Expiration
2040-12-23

AI Technical Summary

Technical Problem

Existing methods face challenges in efficiently generating training images and correct labels for classifiers to identify defective parts, as they require a large number of objects with defective parts and accurate labels, which are difficult to prepare.

Method used

A data generation device and method that includes a region extraction unit to generate second region images and labels from first images, a part label generation unit to create second part labels, a learning image synthesis unit to combine these with normal images, and a correct label generation unit to automatically generate labels for detecting defective regions.

Benefits of technology

Efficiently generates learning images and correct labels for classifiers, reducing the manual effort in creating these components and improving the accuracy of defective part detection.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007576772000001
    Figure 0007576772000001
  • Figure 0007576772000002
    Figure 0007576772000002
  • Figure 0007576772000003
    Figure 0007576772000003
Patent Text Reader

Abstract

To provide a data generation device that efficiently generates data used for learning a discriminator.SOLUTION: A data generation device, which generates data used for learning a discriminator that discriminates whether or not a part to be detected is included in an object from an image obtained by capturing the object, includes: a part image generation unit that generates a second part image including a second part from a first part image obtained by extracting a first part included in a first learning image; a part label generation unit that generates a second part label containing information related to a second part of a second part image from a first part label containing information related to the first part of the first part image; a learning image generation unit that generates a second learning image by synthesizing the second part image with at least a part of an object image; and a correct label generation unit that generates, on the basis of information related to a position where the second part image of the second learning image was synthesized and the second part label, a new correct label indicating a correct answer for detection of the second part in the second learning image.SELECTED DRAWING: Figure 3
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present disclosure relates to a data generation device, a data generation method, and a data generation program. [Background technology]

[0002] There is known a technique in which an image of an object to be inspected is input to a classifier formed by a neural network, and the classifier is made to classify whether or not the object contains a defective part. The classifier is trained using training images of objects containing defective parts, and is thereby able to classify whether or not the object contains a defective part. However, it may be difficult to prepare a large number of objects containing defective parts. In view of this, in Patent Document 1, difference data between an image containing a defective part and an image not containing a defective part is extracted, and the difference data is combined with the image not containing the defective part to create a pseudo training image containing a defective part. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2005-156334 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, training images alone are insufficient for training a classifier, and in addition to the training images, it is necessary to prepare correct answer labels that indicate correct answers for detecting defective portions in the training images.

[0005] An object of the present disclosure is to efficiently generate training images and correct labels used in training a classifier. [Means for solving the problem]

[0006] A data generating device according to one embodiment of the present disclosure is a data generating device that generates data to be used for training a classifier that identifies whether an object includes a target part from an image of the object, and includes: a part image generation unit that generates a second part image including a second part from a first part image obtained by extracting a first part included in a first training image; a part label generation unit that generates a second part label including information about the second part of the second part image from a first part label including information about the first part of the first part image; a training image generation unit that generates a second training image by synthesizing the second part image with at least a portion of an image of the object; and a correct label generation unit that generates a new correct label indicating the correct detection of the second part in the second training image based on information about the position where the second part image is synthesized in the second training image and the second part label.

[0007] A data generation method according to one embodiment of the present disclosure is a data generation method for generating data to be used for training a classifier that identifies whether an object includes a target part from an image of the object, the data generation method comprising the steps of: generating a second part image including a second part from a first part image obtained by extracting a first part included in a first training image; generating a second part label including information about the second part of the second part image from a first part label including information about the first part of the first part image; generating a second training image by synthesizing the second part image with at least a portion of an image of the object; and generating a new correct answer label indicating the correct detection of the second part in the second training image based on information about the position where the second part image was synthesized in the second training image and the second part label.

[0008] A data generation program according to one embodiment of the present disclosure is a data generation program that generates data to be used for training a classifier that identifies whether an object includes a target part from an image of the object, and causes a computer to execute the following steps: generate a second part image including a second part from a first part image obtained by extracting a first part included in a first training image; generate a second part label including information about the second part of the second part image from a first part label including information about the first part of the first part image; generate a second training image by synthesizing the second part image with at least a portion of an image of the object; and generate a new correct answer label indicating the correct detection of the second part in the second training image based on information about the position where the second part image in the second training image is synthesized and the second part label.

[0009] Furthermore, these comprehensive or specific aspects may be realized by a system, an apparatus, a method, an integrated circuit, a computer program, or a recording medium, or may be realized by any combination of a system, an apparatus, a method, an integrated circuit, a computer program, and a recording medium. Effect of the Invention

[0010] According to the present disclosure, training images and correct labels used for training a classifier can be efficiently generated. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 is a schematic diagram illustrating a configuration example of an inspection system according to an embodiment; [Figure 2A] FIG. 13 is a diagram for explaining an example in which part position information included in a correct label indicates a rectangular frame; [Figure 2B] FIG. 13 is a diagram for explaining an example in which body part position information included in a correct label indicates a segment region; [Diagram 3] FIG. 1 is a block diagram showing an example of the configuration of a data generating device and a learning device according to an embodiment; [Figure 4]FIG. 1 is a diagram for explaining a process of generating second learning data from first learning data; [Diagram 5] A flowchart showing an example of processing executed by the data generating device and the learning device. [Figure 6] FIG. 2 is a diagram illustrating an example of a hardware configuration of a data generating device and a learning device according to the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, the embodiments of the present disclosure will be described in detail with reference to the drawings as appropriate. However, more detailed description than necessary may be omitted. For example, detailed description of already well-known matters and duplicated description of substantially the same configuration may be omitted. This is to avoid the following description becoming unnecessarily redundant and to facilitate understanding by those skilled in the art. Note that the accompanying drawings and the following description are provided to enable those skilled in the art to fully understand the present disclosure, and are not intended to limit the subject matter described in the claims.

[0013] (One embodiment) <Inspection system configuration> 1 is a schematic diagram showing an example of the configuration of an inspection system 10 according to an embodiment. Next, the configuration of the inspection system 10 according to the embodiment will be described with reference to FIG.

[0014] The inspection system 10 includes an imaging device 11, an inspection device 12, a data generating device 100, and a learning device 200. The imaging device 11 and the inspection device 12 may be configured as a single device. The data generating device 100 and the learning device 200 may be configured as a single device.

[0015] The imaging device 11 is connected to the inspection device 12 by wired communication or wireless communication. The imaging device 11 captures an object 1 to be inspected and generates an object image. The imaging device 11 transmits the generated object image to the inspection device 12. Examples of the object 1 to be inspected include parts used in the manufacture of products, such as automobile bodies, metal processed parts such as gears and screws, and electric parts such as IC chips and printed circuit boards.

[0016] The inspection device 12 receives an object image from the imaging device 11. The inspection device 12 uses the classifier 300 to detect defective parts 2 present in the object image. Examples of defective parts 2 include scratches, uneven color, stains, chips, cracks, burrs, foreign matter, faint printing, and misaligned printing, which are not present in a normal object 1. Note that, although the present embodiment describes a case where a part considered to be defective is detected, the detection target is not limited to a part considered to be defective. For example, the detection target may be any part as long as it has a specific characteristic.

[0017] The classifier 300 may be a machine learning model that detects whether or not a defective portion 2 is included in an object image. When an object image is input, the classifier 300 may be configured to output detection result information indicating whether or not the object image includes a defective portion 2. Furthermore, when the object image includes a defective portion 2, the classifier 300 may be configured to output information indicating the position of the defective portion 2 in the object image (hereinafter referred to as portion position information) as the detection result information. Furthermore, when the object image includes a defective portion 2, the classifier 300 may be configured to output information indicating the type of the defective portion 2 (hereinafter referred to as portion type information) as the detection result information. Examples of the types of the defective portion 2 include scratches, uneven color, stains, chips, cracks, burrs, foreign matter, faint printing, and misaligned printing.

[0018] The classifier 300 may be configured as a CNN (Convolutional Neural Network), which is an example of a neural network or a deep neural network. Note that the classifier 300 is not limited to a CNN, and may be configured as, for example, a RNN (Recurrent Neural Network) or an SVM (Support Vector Machine). The classifier 300 may be replaced with other terms such as a learning device, a classifier, or an AI (Artificial intelligence).

[0019] The learning device 200 uses a plurality of learning data to train the classifier 300. The learning data includes a learning image corresponding to an object image including a defective portion 2, and a correct label indicating the position of the defective portion 2 in the learning image. That is, the learning image and the correct label are associated with each other by the learning data. The learning data may be interpreted as teacher data. For example, the learning device 200 may input a learning image to the classifier 300 and obtain detection result information output from the classifier 300. Then, the learning device 200 may compare the detection result information with a correct label associated with the input learning image, and adjust each parameter of the classifier 300 so that a deviation (difference) between the detection result information and the correct label becomes small. The learning device 200 may improve the accuracy of the classifier 300 in detecting the defective portion 2 by adjusting the parameters of the classifier 300 using a plurality of learning data. Details of the learning device 200 will be described later.

[0020] Here, an example of a learning image and a correct label will be described. Fig. 2A is a diagram for explaining an example of a case where part position information included in a correct label indicates a rectangular frame. Fig. 2B is a diagram for explaining an example of a case where part position information included in a correct label indicates a segment area. The segment area may be an area obtained by segmenting a defective part 2 in an object image.

[0021] 2A, when the part position information is information indicating a rectangular frame 141, the part position information may be information indicating the coordinate points of the upper left vertex and the lower right vertex of the rectangular frame 141 surrounding the defective part 2 in the learning image. In this case, the part position information can also be said to be information indicating the position and shape (e.g., a horizontally long rectangle or a vertically long rectangle, etc.) of the frame 141 including the defective part 2 and the type of defect (part type information). Note that the frame surrounding the defective part 2 is not limited to a rectangular frame and may be, for example, an ellipse or a polygon.

[0022] As shown in FIG. 2B, when the part position information is information indicating the segment area 142, the part position information may be a binary image in which the pixels of the defective part 2 in the learning image are filled in. In this case, the part position information can also be said to be information indicating the position and shape of the defective part 2 and the type of defect (part type information). In the case of segmentation, information indicating the type of defect may be added using pixel color information. The segment area 142 is not limited to being represented as a binary image, and may be, for example, an image with three or more values, an image indicating the outline of the segment area, or the like.

[0023] In this embodiment, a case will be described in which the part position information indicates a segment region 142 as shown in Fig. 2B. However, this embodiment can also be implemented in the case in which the part position information indicates a rectangular frame 141 as shown in Fig. 2A.

[0024] The data generating device 100 generates training data used for training the classifier 300 in the training device 200. The data generating device 100 will be described in detail later.

[0025] <Details of the data generation device and learning device> Fig. 3 is a block diagram showing an example of the configuration of the data generating device 100 and the learning device 200 according to an embodiment. Fig. 4 is a diagram for explaining a process of generating second learning data from first learning data. Next, the configurations of the data generating device 100 and the learning device 200 according to an embodiment will be described in detail with reference to Figs. 3 and 4.

[0026] <<Data Generation Device>> As shown in FIG. 3, data generating device 100 includes first training image input unit 101, label condition setting unit 102, re-learning data input unit 103, first correct label generation unit 104, first training data storage unit 105, part extraction unit 106, first part data storage unit 107, first part data selection unit 108, second part image generation unit 109, second part label generation unit 110, normal image input unit 111, second training image generation unit 112, second correct label generation unit 113, second training data storage unit 114, and training data addition unit 115. The first training image input unit 101, the label condition setting unit 102, the re-learning data input unit 103, the first correct label generation unit 104, the part extraction unit 106, the first part data selection unit 108, the second part image generation unit 109, the second part label generation unit 110, the normal image input unit 111, the second training image generation unit 112, the second correct label generation unit 113, and the training data addition unit 115 may be functions realized by the processor 1001 shown in FIG. 6 cooperating with the memory 1002 to execute a computer program. Therefore, in this embodiment, the process described mainly with these functions can be interpreted as a process mainly with the processor 1001. The first training data storage unit 105, the first part data storage unit 107, and the second training data storage unit 114 may be constructed in at least one of the memory 1002 and the storage 1003 shown in FIG. 6.

[0027] The first training image input unit 101 accepts input of a first training image. For example, a user operates an input device 1004 (see FIG. 6 ) to input the first training image to the first training image input unit 101. The first training image input unit 101 outputs the input first training image to the first correct answer label generation unit 104.

[0028] The label condition setting unit 102 accepts input of conditions (label conditions) for generating the first correct label. For example, the user operates the input device 1004 to input to the label condition setting unit 102 whether the part position information included in the first correct label is to be a rectangular frame 141 or a segment area 142. For example, the user operates the input device 1004 (see FIG. 6 ) to input to the label condition setting unit 102 which type the part type information included in the first correct label is to be. The label condition setting unit 102 sets the input label condition in the first correct label generation unit 104.

[0029] The re-learning data input unit 103 accepts input of re-learning data. The re-learning data may be input from the re-learning data extraction unit 208 described later. The re-learning data input unit 103 outputs a correct label included in the input re-learning data to the first correct label generation unit 104 as a first correct label. In addition, the re-learning data input unit 103 outputs an evaluation image included in the input re-learning data to the first correct label generation unit 104 as a first learning image.

[0030] The first correct label generation unit 104 generates a first correct label to be associated with the first training image input from the first training image input unit 101 or the re-training data input unit 103, based on the label condition set by the label condition setting unit 102. The first correct label may be interpreted as the original correct label.

[0031] For example, when a label condition that sets the part position information as rectangular frame 141 is input, first correct label generation unit 104 displays the first training image on output device 1005 (see FIG. 6) and has the user surround defective part 2 in the first training image with a rectangular frame. First correct label generation unit 104 generates information indicating the coordinates of the upper left vertex and the lower right vertex of rectangular frame 141 surrounded by the user as part position information. First correct label generation unit 104 generates a first correct label including the part position information.

[0032] For example, when a label condition is set such that the part position information is the segment area 142, the first correct label generation unit 104 displays the first training image on the output device 1005 (see FIG. 6) and has the user fill in the defective part 2 in the first training image. The first correct label generation unit 104 generates binary image data indicating the area filled in by the user as part position information.

[0033] For example, when a label condition is set in which the region type information is "scratch", the first correct label generation unit 104 generates region type information indicating "scratch".

[0034] The first label generation unit generates first training data by associating the first correct label including the body part position information and body part type information generated above with the first training image, and outputs the first training data to the first training data storage unit 105.

[0035] The first training data storage unit 105 stores the first training data input from the first correct label generation unit 104. That is, the first training data storage unit 105 stores the first training images and the first correct labels associated with each other by the first training data. In addition, the first training data storage unit 105 stores the second training data input from the training data addition unit 115 as the first training data, as described later.

[0036] Part extraction section 106 acquires the first learning data from first learning data storage section 105, and performs the following process.

[0037] As shown in Fig. 4, part extraction unit 106 extracts a region including a defective part 2 in a first learning image from the first learning image included in the first learning data. For example, part extraction unit 106 displays a list of first learning data stored in first learning data storage unit 105 on output device 1005 (see Fig. 6) and allows a user to select one piece of first learning data. Then, part extraction unit 106 extracts an image of a rectangular frame indicated by a first correct label included in the selected first learning data from the first learning image, and sets it as a first part image.

[0038] Furthermore, part extraction section 106 extracts information on the region corresponding to the first part image extracted above as a first part label from the first correct answer label included in the first learning data, as shown in Fig. 4. Note that part extraction section 106 may extract part type information from the first correct answer label and include it in the first part label.

[0039] For example, when the defective part information of the first correct answer label is binary image data indicating segment region 142, part extraction section 106 extracts an image of a region corresponding to the first part image from the binary image data as the first part label. In this case, the first part label may be binary image data.

[0040] For example, when the defective part information of the first correct label is information indicating the coordinates of the upper left vertex and the lower right vertex of rectangular frame 141, part extraction section 106 extracts, as the first part label, the relative coordinates of the lower right vertex with respect to the coordinates of the upper left vertex of rectangular frame 141. In this case, the first part label may be information indicating the relative coordinates of the lower right vertex with respect to the upper left vertex of the rectangular frame.

[0041] Part extraction section 106 generates first part data by associating the first part image and the first part label generated above, and outputs the first part data to first part data storage section 107.

[0042] In this way, when the user performs an operation to extract a first region image from a first learning image, region extraction unit 106 automatically extracts a first region label to be associated with the first region image from the first answer label associated with the first learning image. This eliminates the need for the user to manually create a first region label to be associated with the first region image, improving the efficiency of generating second learning data.

[0043] First region data storage section 107 stores the first region data input from region extraction section .

[0044] First part data selection unit 108 selects first part data from first part data storage unit 107. For example, first part data selection unit 108 displays a plurality of first part images included in first part data storage unit 107 on output device 1005 (see FIG. 6 ) and allows the user to select one first part image. First part data selection unit 108 outputs the selected first part image to second part image generation unit 109. In addition, first part data selection unit 108 outputs a first part label associated with the selected first part image to second part label generation unit 110.

[0045] The second part image generating unit 109 generates a second part image from the first part image input from the first part data selecting unit 108. For example, the second part image generating unit 109 displays the first part image on the output device 1005 (see FIG. 6) and accepts a transformation operation on the first part image from the user. Examples of the transformation operation include image enlargement, reduction, rotation, inversion, distortion, trimming, color change, brightness change, saturation change, and contrast change. The second part image generating unit 109 transforms the first part image based on the transformation operation by the user to generate a second part image. For example, as shown in FIG. 4, when a rotation operation is performed on the first part image, the second part image generating unit 109 rotates the first part image to generate a second part image. Note that, when a transformation operation is not performed on the first part image, the second part image generating unit 109 may use the first part image as it is as the second part image without transforming it. Second partial region image generating section 109 outputs the generated second partial region image to second training image generating section 112.

[0046] The second region label generation unit 110 converts the first region label based on the first region label input from the first region data selection unit 108 and the transformation from the first region image to the second region image in the second region image generation unit 109, to generate a second region label.

[0047] For example, as shown in FIG. 4, when second region image generation unit 109 rotates a first region image to generate a second region image, second region label generation unit 110 similarly rotates the first region label (e.g., a binary image or a rectangular frame) to generate a second region label.

[0048] For example, when the second part image generation unit 109 inverts the first part image to generate the second part image, the second part label generation unit 110 similarly inverts the first part label (e.g., a binary image or a rectangular frame) to generate the second part label.

[0049] For example, when the second part image generation unit 109 enlarges (or reduces) the first part image by a certain ratio to generate the second part image, the second part label generation unit 110 enlarges (or reduces) the first part label (e.g., a binary image or a rectangular frame) by the same ratio to generate the second part label.

[0050] For example, when second region image generation unit 109 crops a first region image to generate a second region image, second region label generation unit 110 similarly crops the first region label to generate a second region label.

[0051] Furthermore, second region label generation section 110 may include the region type information included in the first region label in the second region label.

[0052] Second region label generation unit 110 outputs the generated second region labels to second correct answer label generation unit 113.

[0053] In this way, when the user performs a transformation operation on the first part image to generate a second part image, second part label generation unit 110 performs a similar transformation on the first part label corresponding to the first part image to automatically generate a second part label. This saves the user the trouble of manually creating a second part label to be associated with the second part image, improving the efficiency of generating second learning data.

[0054] The normal image input unit 111 accepts input of an image (called a normal image) of an object that does not include defects (i.e., a normal object). For example, a user operates the input device 1004 to input a normal image to the normal image input unit 111. The normal image input unit 111 outputs the input normal image to the second training image generation unit 112. However, the normal image may be an image that includes a defective portion 2. In this case, the second training image generation unit 112 may combine the second portion image with the normal image that includes the defective portion 2.

[0055] The second training image generating unit 112 synthesizes the normal image input from the normal image input unit 111 and the second part image input from the second part image generating unit 109 to generate a second training image. For example, the second training image generating unit 112 displays the normal image on the output device 1005 (see FIG. 6 ) and allows the user to input a position at which the second part image is to be synthesized with respect to the normal image. Then, the second training image generating unit 112 synthesizes the second part image with respect to the normal image at the position input by the user to generate a second training image. The second training image may be interpreted as an object image including a pseudo defective part. The second training image generating unit 112 may synthesize the normal image and the second part image by at least one of alpha blending and Poisson image synthesis, or by another method. The second training image generating unit 112 outputs the generated second training image to the second correct answer label generating unit 113.

[0056] The second correct label generation unit 113 generates a second correct label based on the second region label input from the second region label generation unit 110 and the second training image input from the second training image generation unit 112. For example, the second correct label generation unit 113 generates a second correct label by adding information indicating a position where the second region image is synthesized in the second training image to the second region label. The added information may be region position information in the second correct label. The second correct label may be read as a new correct label.

[0057] Furthermore, second correct label generation unit 113 may add the region type information included in the second region label to the second correct label.

[0058] The second correct label generation unit 113 generates second training data by associating the generated second correct label with the second training image. The second correct label generation unit 113 outputs the generated second training data to the second training data storage unit 114.

[0059] In this way, when the user generates the second training image by synthesizing the second region image with the normal image, the second correct label generation unit 113 automatically generates a second correct label by adding information indicating the position where the second region image is synthesized to the second region label. This eliminates the need for the user to manually create a second correct label to be associated with the second training image, improving the efficiency of generating the second training data.

[0060] Moreover, the second correct label generating unit 113 automatically associates the second correct label thus automatically generated with the second learning image. Therefore, the user does not need to manually create a correct label for the second learning image by specifying an area of ​​the second part image included in the second learning image. That is, according to the present embodiment, the efficiency of generating the second learning data is improved.

[0061] The second training data storage unit 114 stores the second training data input from the second correct label generation unit 113.

[0062] The learning data adding unit 115 outputs the second learning data stored in the second learning data storage unit 114 to the first learning data storage unit 105. As a result, the second learning data is stored in the first learning data storage unit 105, as described above.

[0063] <<Learning device configuration>> As shown in FIG. 3, the learning device 200 includes a learning condition setting unit 201, a classifier learning unit 202, a classifier storage unit 203, an evaluation condition setting unit 204, an evaluation image input unit 205, a classifier evaluation unit 206, an evaluation result storage unit 207, and a re-learning data extraction unit 208. The learning condition setting unit 201, the classifier learning unit 202, the evaluation condition setting unit 204, the evaluation image input unit 205, the classifier evaluation unit 206, and the re-learning data extraction unit 208 may be functions realized by the processor 1001 shown in FIG. 6 operating in cooperation with the memory 1002 to execute a computer program. Therefore, in this embodiment, the process described with these functions as the main components can be interpreted as a process mainly performed by the processor 1001. The classifier storage unit 203 and the evaluation result storage unit 207 may be constructed in at least one of the memory 1002 and the storage 1003 shown in FIG. 6.

[0064] The learning condition setting unit 201 accepts selection of first learning data to be used for training the classifier 300. For example, the learning condition setting unit 201 displays the first learning data stored in the first learning data storage unit 105 on the output device 1005 (see FIG. 6 ) and allows the user to select the first learning data to be used for training the classifier 300. In addition, the learning condition setting unit 201 accepts input of learning conditions for the classifier 300. Examples of the learning conditions include the number of epochs, a learning rate, and a batch size. For example, the learning condition setting unit 201 allows the user to input at least one of the number of epochs, the learning rate, and the batch size. The learning condition setting unit 201 sets the identification information of the selected first learning data and the input learning conditions in the classifier training unit 202.

[0065] The classifier training unit 202 acquires first training data corresponding to the identification information of the first training data set by the learning condition setting unit 201 from the first training data storage unit 105. The classifier training unit 202 uses the acquired first training data to train the classifier 300. At that time, the classifier training unit 202 trains the classifier 300 based on the learning conditions set by the learning condition setting unit 201.

[0066] When the classifier 300 is configured as a CNN, the classifier training unit 202 may train the classifier 300 by, for example, the following processes (A1) to (A3). (A1) The classifier training unit 202 inputs a first training image included in first training data to the classifier 300 (CNN), and obtains detection result information output from the classifier 300 (CNN). (A2) The classifier training unit 202 compares the detection result information with a first correct label included in the first training data, and adjusts each parameter of the classifier 300 (CNN) so as to reduce the deviation (difference) between the detection result information and the first correct label. (A3) The classifier training unit 202 performs the above processes (A1) and (A2) on each piece of first training data acquired from the first training data storage unit 105.

[0067] The classifier learning unit 202 outputs the classifier 300 that has completed learning to the classifier storage unit 203 .

[0068] The classifier storage unit 203 stores the classifier 300 input from the classifier learning unit 202 .

[0069] The evaluation condition setting unit 204 accepts the selection of the classifier 300 to be evaluated. For example, the evaluation condition setting unit 204 displays a list of the classifiers 300 stored in the classifier storage unit 203 on the output device 1005 (see FIG. 6 ) and allows the user to select the classifier 300 to be evaluated. In addition, the evaluation condition setting unit 204 accepts input of evaluation conditions for the classifier 300. Examples of the evaluation conditions include an evaluation threshold for determining whether the detection result information output from the classifier 300 is correct or not, a threshold for the number of detections, area or dimensions obtained from the detection result information, and an output destination of the evaluation result information. For example, the user operates the input device 1004 to input the evaluation conditions to the evaluation condition setting unit 204. The evaluation condition setting unit 204 sets the identification information of the selected classifier 300 to be evaluated and the set evaluation conditions in the classifier evaluation unit 206.

[0070] The evaluation image input unit 205 accepts input of an evaluation image used for evaluating the classifier 300. Here, the evaluation image may be an object image for evaluation different from the first learning image and the second learning image. The evaluation image may or may not include a defective portion 2. For example, a user operates the input device 1004 (see FIG. 6 ) to input the evaluation image to the evaluation image input unit 205. The evaluation image input unit 205 outputs the input evaluation image to the classifier evaluation unit 206.

[0071] The classifier evaluation unit 206 acquires the classifier 300 corresponding to the identification information of the classifier 300 input from the evaluation condition setting unit 204 (i.e., the classifier 300 to be evaluated) from the classifier storage unit 203. The classifier evaluation unit 206 inputs the evaluation image input from the evaluation image input unit 205 to the classifier 300, and obtains detection result information output from the classifier 300. The classifier evaluation unit 206 outputs evaluation result information including the input evaluation image, the output detection result information, and a correct answer label previously associated with the evaluation image to the evaluation result storage unit 207.

[0072] The evaluation result storage unit 207 stores the evaluation result information output from the classifier evaluation unit 206 .

[0073] The re-learning data extraction unit 208 extracts evaluation result information to be used for re-learning from the evaluation result information stored in the evaluation result storage unit 207. For example, the re-learning data extraction unit 208 extracts evaluation result information in which a deviation (difference) between the detection result information and the correct answer label is equal to or greater than an evaluation threshold (i.e., the detection result information is incorrect). The re-learning data extraction unit 208 outputs the extracted evaluation result information to the re-learning data input unit 103 as re-learning data. That is, the re-learning data extraction unit 208 outputs an evaluation image and a correct answer label in which the classifier 300 to be evaluated fails to detect the defective portion 2 to the re-learning data input unit 103. As a result, the evaluation image and the correct answer label in which the classifier 300 to be evaluated fails to detect the defective portion 2 are stored in the first learning data storage unit 105 as a first learning image and a first correct answer label through the re-learning data input unit 103 and the first correct answer label generation unit 104, and are used for re-learning the classifier 300 in the classifier training unit 202. Through this re-learning, the parameters of the classifier 300 are adjusted so that the classifier 300 can detect the defective portion 2 even in evaluation images in which it had initially failed to detect the defective portion 2, thereby improving the accuracy with which the classifier 300 detects the defective portion 2.

[0074] <Processing Executed by Data Generating Device and Learning Device> FIG. 5 is a flowchart showing an example of processing executed by the data generating device 100 and the learning device 200.

[0075] In step S101, the first correct label generation unit 104 generates a first correct label to be associated with the first training image based on a user operation. In addition, the first correct label generation unit 104 generates first training data by associating the first correct label with the first training image, and stores the first training data in the first training data storage unit 105.

[0076] In S102, part extraction unit 106 extracts a first part image from a first training image in a first data storage unit based on a user's operation. Then, part extraction unit 106 extracts a first part label from a first ground truth label associated with the first training image based on the extraction of the first part image for the first training image.

[0077] In S103, second partial region image generating section 109 generates a second partial region image by modifying the first partial region image based on a modification operation by the user.

[0078] In S104, second region label generation section 110 converts the first region label based on the transformation from the first region image to the second region image in S103, and generates a second region label.

[0079] In S105, second training image generating unit 112 generates a second training image by synthesizing the second regional image generated in S103 with the normal image at the position input by the user.

[0080] In S106, second correct label generation unit 113 generates a second correct label based on the second region label generated in S104 and the second training image generated in S105. For example, second correct label generation unit 113 generates the second correct label by adding information indicating the position where the second region image is synthesized with respect to the second training image to the second region label.

[0081] In S107, the second correct label generation unit 113 generates second training data by associating the second correct label generated in S106 with the second training image, and stores the second training data in the second training data storage unit 114.

[0082] In S108, the learning data adding unit 115 stores the second learning data stored in the second learning data storage unit 114 in the first learning data storage unit 105 as the first learning data.

[0083] In S109 , the classifier training unit 202 trains the classifier 300 by using the first training data stored in the first training data storage unit 105 .

[0084] In S110 , the classifier evaluation unit 206 evaluates the classifier 300 using the evaluation image and the correct label, and stores the evaluation result information in the evaluation result storage unit 207 .

[0085] In S111, the re-learning data extraction unit 208 determines whether or not there is evaluation result information to be used for re-learning the classifier 300 among the evaluation result information stored in the evaluation result storage unit 207. Here, the evaluation result information to be used for re-learning the classifier 300 may be evaluation result information in which the detection result information and the correct label are different (i.e., the detection result of the defective portion 2 is incorrect). If it is determined that there is no evaluation result information to be used for re-learning the classifier 300 (S111: NO), this process ends.

[0086] When it is determined that there is evaluation result information to be used for re-learning the classifier 300 (S111: YES), the re-learning data extraction unit 208 outputs the evaluation image and the correct label included in the evaluation result information to be used for re-learning the classifier 300 to the re-learning data input unit 103. Then, the process of S101 is executed. In this case, in S101, the evaluation image and the correct label are stored in the first learning data storage unit 105 as the first learning image and the first correct label, respectively, and the classifier 300 is re-learned in S109, so that the detection accuracy of the defective portion 2 of the classifier 300 can be improved. Also, in S102 to S108, a plurality of second learning images can be derived from the evaluation image in which the detection result of the defective portion 2 was incorrect. Therefore, by re-learning the classifier 300 in S109 using the plurality of second learning data derived from the evaluation image in this way, the detection accuracy of the defective portion 2 of the classifier 300 can be further improved.

[0087] <Hardware configuration> Fig. 6 is a diagram illustrating an example of a hardware configuration of a data generating device 100 and a learning device 200 according to the present disclosure. The data generating device 100 and the learning device 200 may be configured as a computer 1000 illustrated in Fig. 6. In this case, the functional blocks of the data generating device 100 and the learning device 200 described above may be realized by the computer 1000 executing a data generation program, which is an example of a computer program.

[0088] The computer 1000 includes a processor 1001, a memory 1002, a storage 1003, an input device 1004, an output device 1005, a communication device 1006, a graphics processing unit (GPU) 1007, a reading device 1008, and a bus 1009. The processor 1001, the memory 1002, the storage 1003, the input device 1004, the output device 1005, the communication device 1006, the GPU 1007, and the reading device 1008 are connected to the bus 1009, and can transmit and receive data in both directions via the bus 1009.

[0089] The processor 1001 is a device that executes a computer program stored in the memory 1002 and realizes the above-mentioned functional blocks. Examples of the processor 1001 include a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a controller, an LSI (Large Scale Integration), an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), and an FPGA (Field-Programmable Gate Array).

[0090] The memory 1002 is a device that stores computer programs and data operated by the computer 1000. The memory 1002 may include a read-only memory (ROM) and a random access memory (RAM).

[0091] The storage 1003 is configured as a non-volatile storage medium, and is a device that stores computer programs and data handled by the computer 1000. Examples of the storage 1003 include a hard disk drive (HDD), a solid state drive (SSD), and a flash memory.

[0092] The input device 1004 is a device that accepts data to be input to the processor 1001. Examples of the input device 1004 include a keyboard, a mouse, a touch pad, and a microphone.

[0093] The output device 1005 is a device that outputs data generated by the processor 1001. Examples of the output device 1005 include a display and a speaker. For example, a user may input or select desired data by operating a UI (User Interface) displayed on the output device 1005 through the input device 1004.

[0094] The communication device 1006 is a device that transmits and receives data to and from other devices, such as a server or a terminal, via a communication network. The communication device 1006 may include a transmitting unit that transmits data and a receiving unit that receives data. The communication device 1006 may support either wired communication or wireless communication. An example of wired communication is Ethernet (registered trademark). Examples of wireless communication are IEEE802.11, Bluetooth, LTE (Long Term Evolution), 4G, and 5G.

[0095] The GPU 1007 is a device that performs high-speed image rendering processing. The GPU 1007 may be used for AI processing. For example, the GPU 1007 may be used for the learning process of the classifier 300 and / or the detection process by the classifier 300.

[0096] The reader 1008 is a device that reads data from a recording medium such as a digital versatile disk read only memory (DVD-ROM) or a universal serial bus (USB) memory.

[0097] At least one of the functional blocks of the data generating device 100, the learning device 200, and the inspection device 12 may be realized as an LSI, which is an integrated circuit. At least one of the functional blocks may be individually formed into a single chip, or may be formed into a single chip that includes some or all of the functional blocks. Here, the term LSI is used, but depending on the degree of integration, it may be called an IC, a system LSI, a super LSI, or an ultra LSI. Furthermore, if an integrated circuit technology that can replace LSI appears due to the progress of semiconductor technology or a different derived technology, it is natural that the functional blocks may be integrated using that technology.

[0098] Summary of the Disclosure The contents of this disclosure can be expressed as follows.

[0099] <Expression 1> The data generating device 100 generates data to be used for training a classifier 300 that identifies whether or not an object 1 includes a target region 2 from an image of the object 1. The data generating device 100 includes a second region image generating unit 109 that generates a second region image including a second region from a first region image obtained by extracting a first region included in a first training image, a second region label generating unit 110 that generates a second region label including information about the second region of the second region image from a first region label including information about the first region of the first region image, a second training image generating unit 112 that generates a second training image by synthesizing the second region image with at least a portion of the image of the object, and a second correct answer label generating unit 113 that generates a second correct answer label indicating the correct detection of the second region in the second training image based on information about the position where the second region image of the second training image is synthesized and the second region label.

[0100] According to the configuration of Representation 1, a second correct answer label indicating the correct answer for the detection of the second part in the second training image is automatically generated. This eliminates the need for a user to create a second correct answer label associated with the second training image, and improves the efficiency of generating data used for training the classifier 300.

[0101] <Expression 2> In addition, the description of Expression 1 may further include a part extraction unit that extracts information about the first part from a first correct answer label indicating the correct answer to the detection of the first part in the first training image, and generates a first part label.

[0102] According to the configuration of Representation 2, the first region label is automatically generated from the first answer label. This eliminates the need for the user to create the first region label associated with the first region image, and improves the efficiency of generating data used for training classifier 300.

[0103] <Expression 3> Furthermore, in the description of expression 2, the second part image generation unit 109 may deform the first part image to generate the second part image including the second part, and the second part label generation unit 110 may convert information about the first part included in the first part label into information about the second part based on the transformation from the first part image to the second part image, and generate a second part label.

[0104] According to the configuration of Representation 2, a second region label is automatically generated from a first region label based on the transformation from a first region image to a second region image. This eliminates the need for a user to create a second region label associated with the second region image, and improves the efficiency of generating data used for training classifier 300.

[0105] <Expression 4> Furthermore, in the description of Expression 3, the first correct answer label may be an image showing a first part in the first training image as a segment region, the second correct answer label may be an image showing a second part in the second training image as a segment region, the first part label may include an image showing the first part in the first part image as a segment region, and the second part label may include an image showing the second part in the second part image as a segment region.

[0106] <Expression 5> In addition, in the description of Expression 4, the second region label generation section 110 may generate a second region label by converting an image of the first region label in response to the transformation from the first region image to the second region image.

[0107] According to the configurations of Representations 4 and 5, when the first region label is image information indicating the first region as a segment region, the second region label is automatically generated from the first region label. This eliminates the need for the user to create a second region label associated with the second region image, and improves the efficiency of generating data used for training classifier 300.

[0108] <Expression 6> Furthermore, in the description of Expression 2, the first correct answer label is information indicating the position and shape of a frame surrounding a first part in a first training image, the second correct answer label is information indicating the position and shape of a frame surrounding a second part in a second training image, the first part label may include information indicating the shape of the frame surrounding the first part, and the second part label may include information indicating the shape of the frame surrounding the second part.

[0109] <Expression 7> In addition, in the description of Expression 6, the second region label generation unit 110 may generate a second region label by converting information indicating the shape of the frame of the first region label in response to the transformation from the first region image to the second region image.

[0110] According to the configurations of Representations 6 and 7, when the first region label is information indicating the position and shape of a frame surrounding the second region, the second region label is automatically generated from the first region label. This eliminates the need for the user to create a second region label to be associated with the second region image, and improves the efficiency of generating data used for training classifier 300.

[0111] <Expression 8> Furthermore, in the description of any one of expressions 1 to 7, the first correct answer label and the first body part label include body part type information indicating the type of the first body part, and the second body part label generation unit 110 may include the body part type information included in the first body part label in the second body part label.

[0112] According to the configuration of Representation 8, the region type information included in the first region label is carried over to the second region label. This eliminates the need for the user to create region type information for the second region label to be associated with the second region image, and improves the efficiency of generating data used for training classifier 300.

[0113] Although the embodiments have been described above with reference to the accompanying drawings, the present disclosure is not limited to such examples. It is clear that a person skilled in the art can conceive of various modifications, corrections, substitutions, additions, deletions, and equivalents within the scope of the claims, and it is understood that these also belong to the technical scope of the present disclosure. In addition, the components in the above-mentioned embodiments may be arbitrarily combined within the scope of the invention. [Industrial Applicability]

[0114] The techniques disclosed herein are useful, for example, in detecting defective products. [Explanation of symbols]

[0115] 1 object 2 Defective part 10 Inspection System 11 Imaging device 12 Inspection equipment 100 Data generating device 101 First learning image input unit 102 Label condition setting section 103 Re-learning data input section 104 First correct label generation unit 105 First learning data storage unit 106 Part extraction part 107 First part data storage unit 108 First part data selection unit 109 Second local image generating unit 110 Second body part label generation unit 111 Normal image input unit 112 Second learning image generation unit 113 Second correct label generation part 114 Second learning data storage unit 115 Training Data Addition Unit 141 Rectangular Frame 142 Segment Area 200 Learning Device 201 Learning condition setting section 202 Classifier learning unit 203 Classifier storage unit 204 Evaluation condition setting section 205 Evaluation image input unit 206 Classifier evaluation unit 207 Evaluation result storage section 208 Re-learning data extraction part 300 Classifier 1000 Computers 1001 Processor 1002 Memory 1003 Storage 1004 Input Device 1005 Output device 1006 Communication equipment 1007 GPU 1008 Reading device 1009 Bus

Claims

1. 1. A data generating device that generates data used for training a classifier that identifies whether or not a detection target portion is included in an object from an image of the object, the data generating device comprising: a part image generating unit that generates a second part image including a second part from a first part image obtained by extracting a first part included in a first learning image; a region label generation unit that generates a second region label including information about a second region of the second region image from a first region label including information about the first region of the first region image; a training image generating unit that generates a second training image by combining the second portion image with at least a part of the image of the object; a correct label generation unit that generates a new correct label indicating a correct detection of the second region in the second learning image based on information on a position where the second region image in the second learning image is synthesized and the second region label; a region extraction unit that extracts information about the first region from an original correct answer label that indicates a correct answer for the detection of the first region in the first learning image, and generates the first region label; the original correct label is information indicating a position and a shape of a frame surrounding the first region in the first learning image, the new correct label is information indicating a position and a shape of a frame surrounding the second region in the second learning image, the first region label includes information indicating a shape of a frame surrounding the first region; the second region label includes information indicating a shape of a frame surrounding the second region; Data generation device.

2. The body part label generating unit converts information indicating a shape of a frame of the first body part label in response to a transformation from the first body part image to the second body part image, to generate the second body part label. The data generating device according to claim 1 .

3. the local area image generation unit transforms the first local area image to generate the second local area image including the second area; the region label generation unit converts information about the first region included in the first region label into information about the second region based on the transformation from the first region image to the second region image, and generates the second region label. The data generating device according to claim 1 .

4. the original correct label and the first body part label include body part type information indicating a type of the first body part, the region label generation unit includes the region type information included in the first region label in the second region label. The data generating device according to any one of claims 1 to 3.

5. 1. A data generation method for generating data used in training a classifier that identifies whether or not a detection target part is included in an object from an image of the object, the method comprising: generating a second region image including a second region from a first region image obtained by extracting a first region included in a first learning image; generating a second region label including information about a second region of the second region image from a first region label including information about the first region of the first region image; generating a second learning image by synthesizing the second portion image with at least a part of the image of the object; generating a new correct answer label indicating a correct answer for detecting the second region in the second learning image based on information on a position where the second region image in the second learning image is synthesized and the second region label; generating the first region label by extracting information about the first region from an original correct answer label indicating a correct answer for the detection of the first region in the first learning image; the original correct label is information indicating a position and a shape of a frame surrounding the first region in the first learning image, the new correct label is information indicating a position and a shape of a frame surrounding the second region in the second learning image, the first region label includes information indicating a shape of a frame surrounding the first region; the second region label includes information indicating a shape of a frame surrounding the second region; Data generation method.

6. A data generation program for generating data used in training a classifier that identifies whether or not an object includes a detection target portion from an image of the object, the data generating program comprising: generating a second part image including a second part from a first part image obtained by extracting a first part included in a first learning image; generating a second region label including information about a second region of the second region image from a first region label including information about the first region of the first region image; generating a second learning image by synthesizing the second portion image with at least a part of the image of the object; generating a new correct answer label indicating a correct answer for detecting the second region in the second learning image based on information on a position where the second region image in the second learning image is synthesized and the second region label; generating the first region label by extracting information about the first region from an original correct answer label indicating a correct answer for the detection of the first region in the first learning image; the original correct label is information indicating a position and a shape of a frame surrounding the first region in the first learning image, the new correct label is information indicating a position and a shape of a frame surrounding the second region in the second learning image, the first region label includes information indicating a shape of a frame surrounding the first region; the second region label includes information indicating a shape of a frame surrounding the second region; A data generation program that causes a computer to execute the following:

Citation Information

Patent Citations

  • Pseudo defective image automatic creation device and imaging inspection device

    JP2005156334A

  • Data generation device, data generation method, and data generation program

    JP2019114116A

  • Learning data generating device, discrimination model generating device, and program

    JP2020027424A

  • Method for generating data set for learning for detection of obstacle in autonomous driving circumstances and computing device, learning method, and learning device using the same

    JP2020038666A

  • Learning method, learning apparatus, and weld defect detecting apparatus with same learning apparatus

    JP2020144688A