Machine learning device, machine learning method, machine learning program, inspection device, inspection method, and inspection program

Abstract

To provide a machine learning device, a machine learning method, a machine learning program, an inspection device, an inspection method, and an inspection program that can prevent or suppress determination precision of abnormality detection from lowering even when the number of samples of input data is small or when the input data includes a plurality of different patterns mixed.SOLUTION: A machine learning device 100 has an image acquisition part 210, an image division part 220 and a learning part 230. The image acquisition part 210 acquires an image for learning which includes a plurality of different patterns. The image division part 220 divides the image for learning into a plurality of division images by similar patterns. The learning part 230 performs machine learning of generation models by the similar patterns by using the plurality of division images to generate a plurality of learning-completed models.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a machine learning device, a machine learning method, a machine learning program, an inspection device, an inspection method, and an inspection program. [Background technology]

[0002] There are known inspection devices that inspect products for quality using images of the inspection target on a production line. In such inspection devices, for example, feature quantities are extracted from the image of the inspection target, and the quality of the product is judged based on a threshold value that separates good and bad products.

[0003] In this regard, for example, Patent Document 1 listed below discloses an anomaly detection system that determines whether input data such as sensor data and application data is normal or abnormal. This system acquires input data to be subjected to anomaly detection, infers latent variables from the input data based on a latent variable model, and generates restored data from the latent variables based on a joint probability model. Then, it determines whether the input data is normal or abnormal based on the deviation between the input data and the restored data. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2017 / 094267 Summary of the Invention [Problem to be solved by the invention]

[0005] However, the technique of Patent Document 1 has a problem in that the accuracy of anomaly detection decreases when the number of input data samples used for learning is small or when the input data contains a mixture of different patterns.

[0006] The present invention has been made to solve such problems, and aims to provide a machine learning device, a machine learning method, a machine learning program, an inspection device, an inspection method, and an inspection program that can prevent or suppress a decrease in the accuracy of anomaly detection even when the number of input data samples is small or when the input data contains a mixture of multiple different patterns. [Means for solving the problem]

[0007] The above-mentioned problems of the present invention are solved by the following means.

[0008] (1) An image acquisition unit that acquires learning images including a plurality of different patterns, an image division unit that divides the learning images into a plurality of sample images for each similar pattern, and a learning unit that uses the plurality of sample images to machine-learn a generative model for each similar pattern and generate a plurality of trained models. The image dividing unit further includes an area receiving unit that receives areas for dividing the learning image into the sample images. Machine learning device. (2) A machine learning device comprising: an image acquisition unit that acquires training images including a plurality of different patterns; an image division unit that divides the training images into a plurality of sample images for each similar pattern; and a learning unit that uses the plurality of sample images to machine-learn a generative model for each similar pattern and generate a plurality of trained models, wherein the learning unit inputs the sample images and trains using an error backpropagation method so that there is no difference between the reconstructed image output from the trained model and the sample image.

[0009] ( 3 ) the image division unit extracts an image region that matches or has similarity to a specific pattern from the learning image as a similar pattern, or (2) The machine learning device according to claim 1.

[0010] ( 4 ) by user input A plurality of the above The above ( 3 ) The machine learning device described in

[0011] ( 5 ) the image division unit extracts a plurality of image regions that match or have similarity with each other from the learning image as similar patterns, or (2) The machine learning device according to claim 1.

[0012] ( 6) The image segmentation unit extracts image regions of a certain size having matching or similar features as the similar patterns through deep learning, 5 ) The machine learning device described in

[0013] ( 7 ) The method according to any one of (1) to (3), further comprising an image size receiving unit that receives the image size of the sample image. 6 ) The machine learning device described in any one of the above.

[0015] (8) The machine learning device according to any one of (1) to (6) above, wherein the image division unit extracts a repeated pattern in the learning image as the similar pattern.

[0016] (9) The machine learning device described in any one of (1) to (6) and (8) above, wherein the image division unit extracts a pattern that is normal in one area of ​​the learning image and abnormal in another area as a similar pattern.

[0017] (10) The learning image includes an area outside the inspection target where the similar pattern does not exist, The machine learning device according to any one of (1) to (9) above, wherein the image division unit divides only the area in the learning image where the similar pattern exists into a sample image, and does not divide the area outside the inspection target into a sample image.

[0018] (11) The computer, (a) acquiring training images including a plurality of different patterns; The computer (b) dividing the learning image into a plurality of sample images for each similar pattern; The computer and (c) using the plurality of sample images to machine-learn a generative model for each of the similar patterns to generate a plurality of trained models; In the step (b), regions for dividing the learning image into the sample images are received, and the learning image is divided into a plurality of sample images for each similar pattern. Machine learning methods.

[0019] (12) The machine learning method according to (11) above, wherein in step (b), multiple image regions in the learning image that match or are similar to multiple specific patterns are extracted as similar patterns.

[0020] (13) The machine learning method according to (11) or (12) above. The above Machine learning programs for computers to run.

[0021] (14) An inspection device comprising: an image acquisition unit that acquires an image of an inspection object including a plurality of different patterns; an image division unit that divides the entire image into a plurality of divided images for each similar pattern; an image reconstruction unit that reconstructs the divided images to generate a reconstructed image using a trained model generated for each similar pattern; an abnormality determination unit that determines an abnormality in the divided images based on the divided images and the reconstructed image; and an integration unit that integrates the determination results of the abnormality determination unit and outputs the result as an inspection result.

[0022] (15) The inspection device described in (14) above, wherein the image division unit sequentially slides a sliding window that cuts out a portion of the image over the entire area of ​​the image to be inspected to cut out divided images, and the image reconstruction unit reconstructs each of the cut out divided images to generate a reconstructed image.

[0023] (16) The inspection device described in (14) above, wherein the image division unit extracts multiple image areas in the image to be inspected that match or are similar to multiple specific patterns as similar patterns, divides the image into multiple divided images for each similar pattern, and divides image areas in the image to be inspected that do not match or are similar to the multiple specific patterns into multiple divided images based on a predetermined division rule.

[0024] (17) The inspection device according to (16), wherein the abnormality determination unit determines abnormalities in image areas in the inspection image that are not divided by the image division unit based on a predetermined determination rule.

[0025] (18) The computer, (a) acquiring an image of an object to be inspected, the image including a plurality of different patterns; The computer (b) dividing the entire area of ​​the image into a plurality of divided images for each similar pattern; The computer (c) a step of reconstructing the segmented image to generate a reconstructed image using the trained model generated for each of the similar patterns; The computer (d) determining an abnormality in the segmented image based on the segmented image and the reconstructed image; The computer and step (e) of integrating the determination results in step (d) and outputting the integrated results as inspection results.

[0026] (19) The testing method according to (18) above The above A test program to be executed by a computer. [Effects of the Invention]

[0027] According to the present invention, a machine learning device divides a training image into multiple sample images for each similar pattern, and uses the sample images to machine-learn a generative model for each similar pattern, thereby generating multiple trained models. An inspection device divides the entire area of ​​an inspection target image into multiple divided images for each similar pattern, and uses the trained model for each similar pattern to determine anomalies within the divided images. Therefore, even when the number of training images is small or when multiple different patterns are mixed in the training images, it is possible to prevent or suppress a decrease in the accuracy of anomaly detection. [Brief explanation of the drawings]

[0028] [Figure 1] FIG. 1 is a schematic block diagram illustrating a hardware configuration of a machine learning device according to an embodiment. [Figure 2] 2 is a functional block diagram illustrating functions performed by a control unit shown in FIG. 1. FIG. [Figure 3] 2 is a flowchart illustrating a processing procedure of a machine learning method of the machine learning device shown in FIG. 1 . [Figure 4] FIG. 10 is a schematic diagram illustrating an example of a learning image. [Figure 5] FIG. 5 is a schematic diagram showing an enlarged view of a part of the upper left corner of the learning image shown in FIG. [Figure 6A] 10A and 10B are schematic diagrams illustrating specific patterns of learning images. [Figure 6B] 10A and 10B are schematic diagrams illustrating specific patterns of learning images. [Figure 6C] 10A and 10B are schematic diagrams illustrating specific patterns of learning images. [Figure 6D] 10A and 10B are schematic diagrams illustrating specific patterns of learning images. [Figure 7] FIG. 10 is a schematic diagram illustrating a case where an area excluding defects is used for learning a generative model. [Figure 8] FIG. 1 is a schematic block diagram illustrating a hardware configuration of an inspection apparatus according to an embodiment. [Figure 9] 9 is a functional block diagram illustrating functions performed by a control unit shown in FIG. 8. FIG. [Figure 10] 9 is a flowchart illustrating a processing procedure of an inspection method of the inspection device shown in FIG. 8. [Figure 11] FIG. 10 is a schematic diagram illustrating, as a comparative example, a case where an inspection object is divided into a plurality of grid-like regions and photographed. [Figure 12] FIG. 10 is a schematic diagram illustrating, as a comparative example, a case in which a plurality of images are randomly cut out and divided from a learning image. DETAILED DESCRIPTION OF THE INVENTION

[0029] Hereinafter, a machine learning device, a machine learning method, a machine learning program, an inspection device, an inspection method, and an inspection program according to embodiments of the present invention will be described with reference to the drawings. Note that in the drawings, identical elements are designated by the same reference numerals, and duplicated explanations will be omitted.

[0030] FIG. 1 is a schematic block diagram illustrating the hardware configuration of a machine learning device according to an embodiment, and FIG. 2 is a functional block diagram illustrating the functions performed by a control unit.

[0031] <Configuration of machine learning device> The machine learning device of this embodiment divides a training image into multiple sample images for each similar pattern, trains a generative model for each similar pattern, and generates a trained model. A similar pattern is an image region included in the training image that matches or has similarity to a specific pattern.

[0032] 1, the machine learning device 100 includes a control unit 110, a communication unit 120, and an operation display unit 130. These components are connected to each other via a bus 101. The machine learning device 100 may be, for example, a computer such as a personal computer or a server.

[0033] The control unit 110 includes a central processing unit (CPU) 111, a random access memory (RAM) 112, a read only memory (ROM) 113, and an auxiliary storage unit 114.

[0034] The CPU 111 executes programs such as an OS (Operating System) and a machine learning program loaded in the RAM 112, and controls the operation of the machine learning device 100. The machine learning program is stored in advance in the ROM 113 or the auxiliary storage unit 114. The RAM 112 also stores data and the like temporarily generated by the processing of the CPU 111. The ROM 113 stores the programs executed by the CPU 111, as well as data, parameters, and the like used in executing the programs. The auxiliary storage unit 114 includes, for example, a hard disk drive (HDD), a solid state drive (SSD), or the like.

[0035] The communication unit 120 is an interface circuit (for example, a LAN card) for communicating with an external device via a network.

[0036] The operation display unit 130 has an input unit and an output unit. The input unit includes, for example, a keyboard, a mouse, etc., and is used by the user to input characters using the keyboard, mouse, etc., and to enter various instructions (inputs) such as various settings. The output unit includes a display (display device) and displays learning images as input images and an anomaly score map, etc., as output images, which will be described later.

[0037] As shown in FIG. 2, the control unit 110 functions as an image acquisition unit 210, an image division unit 220, a learning unit 230, a pattern reception unit 240, an image size reception unit 250, and an area reception unit 260 by the CPU 111 executing the machine learning program.

[0038] The image acquisition unit 210 acquires training images for use by the learning unit 230 (described later) to train a generative model. The training images are, for example, images of an inspection object. The inspection object is not particularly limited, but examples include circuit boards and components used in industrial products. The inspection includes detecting abnormalities such as creases, bends, chips, scratches, and stains. The training images are stored in advance in, for example, the auxiliary storage unit 114. Furthermore, the image acquisition unit 210 can also acquire training images from an external storage device, server, etc. via a network by cooperating with the communication unit 120.

[0039] In this embodiment, the learning image is an image of a substrate as an inspection target, captured in a direction perpendicular to the substrate surface, and includes a plurality of different patterns (see, for example, FIG. 4). The patterns are formed due to the structure of the substrate, the shapes of the components, etc. For example, the substrate may include a pin-like structure having a plurality of pins formed at equal intervals (for example, first portion 301 in FIG. 5).

[0040] Although not shown, the inspection object is photographed by an imaging device such as a camera. The imaging device transmits image data of the photographed inspection object to the machine learning device 100. Furthermore, the learning images photographed in advance by the imaging device are stored in a storage device external to the machine learning device 100, and the machine learning device 100 can also be configured to sequentially acquire a predetermined number of images of the inspection object stored in the storage device as learning images.

[0041] For example, when the inspection object is a part of an industrial product, the imaging device is installed in the inspection process, captures an image of the imaging range that includes the inspection object, and outputs image data that includes the inspection object. The imaging device outputs, for example, black-and-white or color image data of the inspection object at a predetermined pixel.

[0042] The image dividing unit 220 divides the learning image acquired by the image acquiring unit 210 into multiple sample images for each similar pattern. The image dividing unit 220 extracts, as similar patterns, image regions that match or are similar to a specific pattern from among image regions included in the learning image, for example, by pattern matching processing. The specific pattern can be input by the user, for example, via the operation display unit 130. For example, the user can input a specific pattern by selecting a pattern included in the learning image displayed on the screen of the operation display unit 130 using a mouse or the like. The pattern accepting unit 240 accepts multiple specific patterns input by the user.

[0043] Furthermore, the presence or absence of similarity between a specific pattern and a pattern of a learning image can be determined, for example, by calculating the similarity between the two. The similarity can be calculated, for example, by comparing the feature parameters or feature vectors of the two patterns. If the similarity exceeds a predetermined threshold, the two patterns are determined to be similar, and if the similarity is equal to or less than the predetermined threshold, the two patterns are determined to be dissimilar.

[0044] The specific pattern can also be automatically determined by the image dividing unit 220. The image dividing unit 220 extracts, as similar patterns, a plurality of image regions that match or are similar to each other in the learning image. For example, the image dividing unit 220 extracts, as similar patterns, image regions of a certain size that have matching or similar features by machine learning such as deep learning. The image dividing unit 220 can extract, for example, a certain number of patterns that have matching or similar features as similar patterns.

[0045] The user can also input the size of the sample image through the operation display unit 130. The image size receiving unit 250 receives the size of the sample image input by the user and stores it in the RAM 112.

[0046] The image dividing unit 220 can also determine the size of the sample images so that each sample image contains a similar pattern.

[0047] The learning unit 230 learns a generative model (neural network) for each similar pattern. For example, if there are N similar patterns, the learning unit 230 learns N generative models for each similar pattern and generates a trained model for each of the N similar patterns using a sample image. Each generative model is trained to output a reconstructed image for the sample image. More specifically, the learning unit 230 inputs the sample image and trains using backpropagation to eliminate any difference (loss) between the reconstructed image output from the trained model and the sample image.

[0048] As the generative model, for example, an autoencoder (AE) or a variational autoencoder (VAE) can be suitably used. AE and VAE are well-known techniques, and therefore detailed description thereof will be omitted.

[0049] Alternatively, the user may input a division area for dividing the learning image into a plurality of sample images via the operation display unit 130, and the image dividing unit 220 may divide the learning image into a plurality of sample images based on the division area input by the user. The area receiving unit 260 receives the division area input by the user and stores it in the RAM 112. If the user understands the arrangement of patterns in the learning image, the sample image can be divided effectively.

[0050] <Machine learning method> The processing steps of the machine learning method of the machine learning device of this embodiment will be described in detail below with reference to Figs. 3 to 7. Fig. 3 is a flowchart illustrating the processing steps of the machine learning method of the machine learning device shown in Fig. 1. The processing of the flowchart shown in Fig. 3 is realized by CPU 111 of control unit 110 executing a machine learning program. Fig. 4 is a schematic diagram illustrating an example of a learning image, and Fig. 5 is a schematic diagram of an enlarged portion of the upper left corner of the learning image shown in Fig. 4. Figs. 6A to 6D are schematic diagrams illustrating specific patterns of the learning image, and Fig. 7 is a schematic diagram for explaining a case where an area excluding defects is used to learn a generative model.

[0051] 3, first, a learning image is acquired (step S101). The image acquisition unit 201 acquires the learning image from an external storage device, for example, via the communication unit 120. As shown in FIG. 4, the image acquisition unit 201 acquires, for example, an image 300 of a board as the learning image.

[0052] As shown in FIG. 5, the image 300 of the substrate includes, for example, a first portion 301 to a fourth portion corresponding to a predetermined structure of the substrate. The first portion 301 corresponds to a pin-shaped structure of the substrate. The second portion 302 corresponds to an elliptical structure. The third portion 303 corresponds to an edge of the substrate. The fourth portion 304 corresponds to a circular (hole-shaped) structure. The first portion 301, the second portion 302, and the fourth portion 304 are each arranged in plurality at predetermined intervals.

[0053] Next, the learning image is divided into a plurality of sample images (step S102). The image dividing unit 220 performs a pattern matching process, for example, over the entire area of ​​the substrate image 300, and extracts image regions that match or are similar to each specific pattern as similar patterns.

[0054] The specific pattern is a pattern that includes one or more portions corresponding to a predetermined structure of the substrate. As shown in FIGS. 6A to 6D, the specific patterns may be, for example, first pattern 311 to fourth pattern 314. First pattern 311 includes first portions 301 arranged in 3 rows and 3 columns. Second pattern 312 includes one second portion 302. Third pattern 313 includes a part of third portion 303. Fourth pattern 314 includes one fourth portion 304.

[0055] Although the first pattern 311 includes the first portions 301 of 3 rows and 3 columns, the number of rows and columns of the first portions 301 is not limited to this. The second pattern 312 and the third pattern 313 may each include a plurality of second portions 302 and a plurality of third portions 303.

[0056] For example, the user inputs a specific pattern via the operation display unit 130. The pattern receiving unit 240 receives the specific pattern (for example, the first pattern 311 to the fourth pattern 314, etc.) input by the user.

[0057] First pattern 311 to fourth pattern 314 are patterns that repeatedly appear in board image 300. Image dividing section 220 extracts such repeated patterns in board image 300 as similar patterns.

[0058] The image dividing unit 220 divides the image into a plurality of sample images for each extracted similar pattern. For example, image areas 321 to 324 surrounded by dashed lines in Fig. 5 are shown as sample images. Note that, for convenience of illustration, only a portion of the sample images is shown in Fig. 5, but the entire area of ​​the learning image can be divided into sample images.

[0059] On the other hand, an image region including a defect, such as image region 325, does not match the specific pattern and does not have similarity. In this case, image dividing unit 220 does not divide image region 325 as a sample image, and therefore it is possible to prevent the image region including the defect from being supplied to learning unit 230 as a sample image for learning.

[0060] Furthermore, the image dividing unit 220 can also divide patterns other than the specific pattern into sample images based on a predetermined division rule. For example, an image region including only one first portion does not match the specific pattern and has no similarity to it, but a pattern including one first portion can be treated as a similar pattern, and the image can be divided into sample images for each similar pattern (for example, image region 326 surrounded by a dashed line in FIG. 5). In this way, by applying a predetermined division rule to patterns that do not match or are not similar to the specific pattern, image regions in the learning image that do not include similar patterns can also be divided into sample images.

[0061] Next, a trained model is generated (step S103). The training unit 230 trains a generative model for each similar pattern using a sample image (image areas 321 to 324, 326, etc.) for each similar pattern, and generates a trained model.

[0062] In this embodiment, a good substrate has, from the edge to the inside, a pin-shaped structure, an elliptical structure, a pin-shaped structure, and a circular structure in this order. Therefore, for example, if an elliptical structure is formed where a pin-shaped structure should be formed, or if a pin-shaped structure is formed where an elliptical structure should be formed, the substrate is determined to contain a defect.

[0063] That is, a structure on a substrate (a pattern in a training image) may be normal in one area and abnormal in another area. If a generative model is trained using training images containing abnormal patterns as a whole, false positives may occur in areas with abnormal patterns, which may reduce the accuracy of anomaly detection.

[0064] In contrast, the machine learning device 100 of this embodiment extracts similar patterns even from training images that contain patterns that are normal when present in one area and abnormal when present in another area, and divides the training images into multiple sample images for each similar pattern.The training unit 230 then trains the generative model using multiple sample images that do not contain abnormal patterns, thereby preventing erroneous detection of anomalies and maintaining the accuracy of anomaly detection.

[0065] Furthermore, training images may contain regions that are not the target of inspection. If a generative model is trained using training images that include such regions that are not the target of inspection, as has been done conventionally, false positives may occur in these regions, potentially reducing the accuracy of anomaly detection.

[0066] In contrast, the learning device of this embodiment divides only the areas in the learning image where similar patterns exist into sample images, and does not divide the non-inspection areas where similar patterns do not exist into sample images. Therefore, areas in the learning image that are not to be inspected are excluded from the learning objects of the learning unit 230, preventing erroneous detection of anomalies and maintaining the accuracy of anomaly detection.

[0067] 3, training images are acquired and similar patterns are extracted from the training images. Then, the training images are divided into multiple sample images for each similar pattern, and a generative model is trained for each similar pattern using the multiple sample images to generate a trained model.

[0068] Therefore, even if there are only a small number of training images available for training a generative model, by using sample images obtained by dividing one training image into multiple similar patterns, the number of sample images available for training by the training unit 230 can be increased. Furthermore, since a trained model is generated for each different similar pattern, false detection of anomalies is less likely to occur. Therefore, the performance of the trained model can be improved.

[0069] Furthermore, by reducing the number of patterns in the sample images used for training when training a generative model, training of the generative model can be made easier, thereby improving the performance of the trained model.

[0070] 7, in the machine learning device 100 of this embodiment, even if a learning image acquired by the image acquisition unit 210 contains a defect, it is possible that the region excluding the defect can be used for training a generative model. Even if the learning image contains a defect, the image division unit 220 can divide the image region that does not contain the defect, excluding the image containing the defect, from the learning image into a plurality of sample images. In this case, the training unit 230 can train a generative model using the sample image that does not contain the defect.

[0071] <Configuration of inspection equipment> The inspection device of this embodiment divides the image to be inspected into segmented images for each similar pattern, generates reconstructed images of the segmented images using a trained model for each similar pattern, and determines abnormalities within the segmented images based on the segmented images and the reconstructed images.

[0072] 8, the inspection device 400 includes a control unit 410, a communication unit 420, and an operation display unit 430. These components are connected to each other via a bus 401. The inspection device 400 may be, for example, a computer such as a personal computer or a server.

[0073] The control unit 410 has a CPU 411, a RAM 412, a ROM 413, and an auxiliary storage unit 414. The CPU 411, the RAM 412, the ROM 413, and the auxiliary storage unit 414 have the same configurations as the CPU 111, the RAM 112, the ROM 113, and the auxiliary storage unit 114 of the machine learning device 100, respectively. The communication unit 420 and the operation display unit 430 also have the same configurations as the communication unit 120 and the operation display unit 130 of the machine learning device 100, respectively. To avoid duplication, detailed descriptions of these configurations will be omitted.

[0074] As shown in FIG. 9, the control unit 410 functions as an image acquisition unit 510, an image division unit 520, an image reconstruction unit 530, an abnormality determination unit 540, and an integration unit 550 as a result of the CPU 411 executing the inspection program.

[0075] The image acquisition unit 510 acquires an image of the inspection object by working in cooperation with the communication unit 420. Details of the inspection object and the imaging device that captures the inspection object are as described above.

[0076] The image dividing unit 520 divides the entire area of ​​the image to be inspected into a plurality of divided images (divided image group) for each similar pattern. The divided images included in the divided image group may be divided images of the inspection target of a non-defective product or divided images of the inspection target of a defective product.

[0077] For example, the image segmentation unit 520 may be configured to sequentially slide a sliding window that segments a portion of the image over the entire area of ​​the image to be inspected to segment the image, and then classify the segmented images into similar patterns. The image segmentation unit 520 performs classification by, for example, determining whether the segmented images match or are similar to a specific pattern. The specific pattern may be one of the patterns illustrated in FIGS. 6A to 6D. In classification by the image segmentation unit 520, the segmented image is classified into one of the specific patterns to which it is most similar. Therefore, even if the segmented image includes an abnormal pattern due to a defect or the like, the segmented image may be classified into one of the specific patterns.

[0078] As another method for dividing the image to be inspected, the image dividing unit 520 may be configured to extract image regions that match or are similar to a specific pattern from the image to be inspected as similar patterns, and divide the image into a plurality of divided images for each similar pattern. Therefore, image regions that do not match or are not similar to the specific pattern are not divided.

[0079] Furthermore, image regions in the image to be inspected that do not match any specific patterns and have no similarity can be divided into multiple divided images based on a predetermined division rule. For example, similar to the image division unit 220 of the machine learning device 100, an image region that includes only one first portion 301 does not match any specific patterns and has no similarity, but is treated as a similar pattern that includes one first portion 301, and can be divided into divided images for each similar pattern. In this way, by applying a predetermined division rule to patterns that do not match or are not similar to a specific pattern, even image regions in the inspection image that do not include similar patterns can be divided into divided images.

[0080] The image dividing unit 520 stores the position of each divided image in the image to be inspected (for example, the coordinates of the upper left pixel of the divided image) and the vertical and horizontal sizes in the RAM 412. The position and vertical and horizontal sizes of the divided images are used when the integrating unit 550 integrates the determination results output by the abnormality determining unit 540 to generate the inspection results.

[0081] The image reconstruction unit 530 uses a plurality of trained models (neural networks) trained for each similar pattern to generate a plurality of reconstructed images (a group of reconstructed images) from a plurality of segmented images segmented by the image segmentation unit 520. The trained models are trained using a plurality of sample images in the machine learning device 100. Each trained model of the image reconstruction unit 530 has, for example, a VAE including an encoder 531 and a decoder 532.

[0082] VAE extracts features from the segmented image, extracts only the essential elements of the segmented image, and reconstructs it using the extracted features to generate and output a reconstructed image in which non-essential elements in the segmented image have been removed. In other words, VAE learns only with segmented images that contain normal similar patterns, so it is configured to be able to generate corresponding features for normal segmented images, but cannot generate features corresponding to abnormalities for segmented images in which similar patterns contain defects or other abnormalities, and therefore has no reproducibility.

[0083] In the example shown in FIG. 9, the segmented image includes a first portion 301 corresponding to a pin-shaped structure on the substrate. This pin-shaped structure includes, for example, a defect that occurred during the manufacturing process. When the reconstruction process of the segmented image is executed, a reconstructed image obtained by reconstructing the segmented image is output. The reconstructed image is an image in which only essential elements remain from the segmented image and unnecessary elements have been removed. The pin-shaped structure on the substrate is reconstructed because it is something that the substrate originally has (an essential element). On the other hand, the defect is not reconstructed because it is abnormal (not an essential element).

[0084] In this way, the image reconstructing section 530 generates and outputs a reconstructed image in which non-essential elements in the segmented image have been removed, so the difference between the segmented image and the reconstructed image is greater in the case of a defective product than in the case of a non-defective product.

[0085] The abnormality determination unit 540 performs an abnormality determination based on the plurality of divided images and the plurality of reconstructed images corresponding to the divided images, and outputs a plurality of determination results. When the image dividing unit 520 has obtained divided images for the entire area of ​​the inspection image, the abnormality determination unit 540 performs an abnormality determination for the entire area of ​​the inspection image.

[0086] Furthermore, if there are undivided image regions in the inspection image, the anomaly determination unit 540 can be configured to also perform anomaly determination on these image regions based on a predetermined determination rule. The determination rule can be set experimentally based on the results of performing anomaly determination on multiple undivided image regions, for example. This allows the anomaly determination unit 540 to perform anomaly determination on the entire area of ​​the image to be inspected.

[0087] The integrating unit 550 integrates the multiple determination results from the abnormality determining unit 540 based on the positions and horizontal and vertical sizes of the divided images stored in the RAM 412, and outputs the integrated results as the inspection result.

[0088] <Testing method> Fig. 10 is a flowchart illustrating the processing procedure of the inspection method of the inspection device shown in Fig. 1. The processing of the flowchart shown in the figure is realized by the CPU 411 of the control unit 410 executing an inspection program.

[0089] First, an image of the inspection target is acquired (step S201). Image acquisition unit 510 acquires an image of the inspection target that includes a plurality of different patterns.

[0090] Next, the image to be inspected is divided into a plurality of divided images (step S202). The image dividing unit 520 divides the entire area of ​​the image to be inspected into a plurality of divided images for each similar pattern.

[0091] Next, a reconstructed image of the divided images is generated (step S203). The image reconstructing unit 530 uses the trained model generated for each similar pattern to reconstruct each of the divided images, thereby generating a plurality of reconstructed images.

[0092] Next, an abnormality in the divided images is determined (step S204). The abnormality determination unit 540 outputs a determination result for each divided image based on the divided image and a reconstructed image corresponding to the divided image. For example, the abnormality determination unit 540 calculates the difference (abnormality score) between the divided image and the reconstructed image as the determination result. The calculated abnormality score is stored in the auxiliary storage unit 114.

[0093] Next, the judgment results are integrated and output as the inspection result (step S205). The integration unit 550 integrates multiple abnormality scores based on the position and vertical and horizontal sizes of each divided image stored in RAM 412, and generates and outputs an abnormality score map corresponding to the image to be inspected. For example, the integration unit 550 displays the generated abnormality score map on the display device of the operation and display unit 430. Since the abnormality score is represented by the difference between the divided image and the corresponding reconstructed image, in the abnormality score map displayed on the display device, parts with abnormalities have a large difference and are displayed brighter than parts without abnormalities. This allows the user to easily identify abnormal locations in the inspection target.

[0094] <Comparative Example> FIG. 11 is a schematic diagram illustrating, as a comparative example, a case where an inspection object is divided into a plurality of grid-like regions and photographed. When photographing an inspection object to obtain a sample image for learning, if the size of the inspection object is large, the photographed image may not have sufficient resolution. In response to this, for example, a method of photographing the inspection object divided into a plurality of regions can be considered to avoid a decrease in resolution. The same figure illustrates an example where the inspection object is photographed divided into ten grid-like regions 331 to 340. In this case, each of the image regions 331 to 340 becomes a sample image for learning.

[0095] In this way, by photographing the inspection object in multiple regions, it is possible to increase the number of sample images for training the generative model while preventing a decrease in the resolution of the inspection object image. However, with this method, there is a high possibility that multiple different patterns will be mixed in the same sample image, and even if the generative model is trained, the accuracy of the trained model may not be sufficiently improved. As a result, the image of the inspection object will be reconstructed (estimated) using a trained model with insufficient accuracy, which can lead to the problem of false detection.

[0096] Furthermore, since the range of the image of the inspection object is wide, the image capture range includes multiple structures on the board, such as pin-shaped structures, elliptical structures, structures at the edge of the board, circular structures, etc., and the learning image may include multiple patterns corresponding to these structures. For patterns that appear less frequently among these, learning progresses slowly, and there is a possibility of false detection.

[0097] 12 is a schematic diagram illustrating, as a comparative example, a case in which multiple images are randomly cut out from a training image and divided. By randomly cutting out multiple images from a training image, the number of divided images for training a generative model can be increased, but the number of patterns contained in the cut-out images may vary. As a result, the increased number of sample images for training may not be effective for training a generative model.

[0098] The machine learning device, machine learning method, machine learning program, inspection device, inspection method, and inspection program of the present embodiment described above can achieve the following effects.

[0099] The machine learning device 100 divides a training image into multiple sample images for each similar pattern, and uses the sample images to machine-learn a generative model for each similar pattern, generating multiple trained models. The inspection device 400 divides the entire area of ​​the image to be inspected into multiple divided images for each similar pattern, and uses the trained model for each similar pattern to determine abnormalities in the divided images. Therefore, even when the number of training images is small or when multiple different patterns are mixed in the training images, it is possible to prevent or suppress a decrease in the accuracy of anomaly detection.

[0100] The machine learning device, machine learning method, machine learning program, inspection device, inspection method, and inspection program described above are merely examples of the main configurations described in the description of the features of the above-described embodiments, and are not limited to the above configurations and may be modified in various ways within the scope of the claims. Furthermore, configurations provided in general inspection devices, etc., are not excluded.

[0101] For example, in the above-described embodiment, the case where the specific pattern is input by the user has been exemplified, but the specific pattern may be stored in advance in the auxiliary storage unit 114 or the like.

[0102] In addition, some steps may be omitted from the above-described flowcharts, other steps may be added, some of the steps may be executed simultaneously, or one step may be divided into multiple steps and executed.

[0103] The means and methods for performing various processes in the above-described device can be realized by either a dedicated hardware circuit or a programmed computer. The above-described program may be provided by a computer-readable recording medium such as a USB memory or a DVD (Digital Versatile Disc)-ROM, or may be provided online via a network such as the Internet. In this case, the program recorded on the computer-readable recording medium is typically transferred and stored in a storage unit such as a hard disk. The above-described program may be provided as standalone application software, or may be incorporated as a function into the software of a device such as a machine learning device or an inspection device. [Explanation of symbols]

[0104] 100 machine learning devices, 110 control section, 111 CPUs, 112 RAM, 113 ROMs, 114 Auxiliary storage, 120 Communications Department, 130 Operation display section, 210 image acquisition unit, 220 Image segmentation unit, 230 Learning Department, 240 pattern reception unit, 250 image size reception section, 260 area reception unit, 400 inspection equipment, 410 control section, 411 CPUs, 412 RAM, 413 ROM, 414 Auxiliary storage, 420 Communications Department, 430 Operation display section, 510 image acquisition unit, 520 image segmentation unit, 530 image reconstruction unit, 531 encoder, 532 decoder, 540 Abnormality determination section, 550 Integrated Department.

Claims

1. an image acquisition unit that acquires learning images including a plurality of different patterns; an image dividing unit that divides the learning image into a plurality of sample images for each similar pattern; a learning unit that uses the plurality of sample images to machine-learn a generative model for each of the similar patterns and generates a plurality of trained models; The image dividing unit further includes a region receiving unit that receives regions for dividing the learning image into the sample image.

2. an image acquisition unit that acquires learning images including a plurality of different patterns; an image dividing unit that divides the learning image into a plurality of sample images for each similar pattern; a learning unit that uses the plurality of sample images to machine-learn a generative model for each of the similar patterns and generates a plurality of trained models; The learning unit inputs the sample image and learns using a backpropagation algorithm so that there is no difference between the reconstructed image output from the trained model and the sample image.

3. The machine learning device according to claim 1 , wherein the image segmentation unit extracts, from the learning image, an image region that matches or has similarity to a specific pattern as a similar pattern.

4. The machine learning device according to claim 3 , further comprising a pattern accepting unit that accepts a plurality of the specific patterns as input by a user.

5. The machine learning device according to claim 1 , wherein the image segmentation unit extracts, from the learning image, a plurality of image regions that match or have similarity with each other as similar patterns.

6. The machine learning device according to claim 5 , wherein the image segmentation unit extracts, as the similar patterns, image regions of a certain size having matching or similar features through deep learning.

7. The machine learning device according to claim 1 , further comprising an image size receiving unit that receives an image size of the sample image.

8. The machine learning device according to claim 1 , wherein the image dividing unit extracts a repeated pattern in the learning image as the similar pattern.

9. The machine learning device according to any one of claims 1 to 6 and 8, wherein the image segmentation unit extracts, as similar patterns, patterns that are normal in one area of ​​the learning image and abnormal in another area.

10. the learning image includes an area outside the inspection target where the similar pattern does not exist, 10. The machine learning device according to claim 1, wherein the image dividing unit divides only an area in the learning image where the similar pattern exists into a sample image, and does not divide an area outside the inspection target into a sample image.

11. A step (a) in which a computer acquires training images including a plurality of different patterns; (b) dividing the learning image into a plurality of sample images for each similar pattern by the computer; and (c) a step in which the computer uses the plurality of sample images to machine-learn a generative model for each of the similar patterns, thereby generating a plurality of trained models; In the step (b), a machine learning method is provided, which receives regions for dividing the learning image into the sample images, and divides the learning image into a plurality of sample images for each similar pattern.

12. The machine learning method according to claim 11 , wherein in the step (b), a plurality of image regions that match or have similarity to a plurality of specific patterns are extracted from the learning image as similar patterns.

13. A machine learning program for causing a computer to execute the machine learning method according to claim 11 or 12.

14. an image acquisition unit that acquires an image of an inspection object including a plurality of different patterns; an image dividing unit that divides the entire area of ​​the image into a plurality of divided images for each similar pattern; an image reconstruction unit that reconstructs the divided images to generate a reconstructed image using a trained model generated for each of the similar patterns; an abnormality determination unit that determines an abnormality in the divided image based on the divided image and the reconstructed image; an integration unit that integrates the determination results of the abnormality determination units and outputs the integrated results as an inspection result.

15. the image dividing unit sequentially slides a sliding window that cuts out a partial area of ​​the image over the entire area of ​​the image to be inspected to cut out divided images; The inspection apparatus according to claim 14 , wherein the image reconstruction unit reconstructs each of the cut-out divided images to generate a reconstructed image.

16. 15. The inspection device according to claim 14, wherein the image dividing unit extracts, from the image to be inspected, a plurality of image regions that match or are similar to a plurality of specific patterns as similar patterns, divides the image to be inspected into a plurality of divided images for each of the similar patterns, and divides, from the image to be inspected, image regions that do not match or are similar to the plurality of specific patterns into a plurality of divided images based on a predetermined division rule.

17. The inspection device according to claim 16 , wherein the abnormality determination unit determines whether an image region in the image to be inspected that is not divided by the image division unit is abnormal based on a predetermined determination rule.

18. A method comprising: (a) acquiring, by a computer, an image of an object to be inspected, the image including a plurality of different patterns; (b) dividing the entire area of ​​the image into a plurality of divided images for each similar pattern by the computer; (c) a step in which the computer reconstructs the segmented image using the trained model generated for each of the similar patterns to generate a reconstructed image; (d) determining an abnormality in the segmented image based on the segmented image and the reconstructed image by the computer; and (e) a step in which the computer integrates the determination results in the step (d) and outputs the integrated results as an inspection result.

19. An inspection program for causing the computer to execute the inspection method according to claim 18.

Images