Defect detection learning device, image inspection device, image inspection system, defect detection learning method, and program

Abstract

To provide a defect detection learning device or the like having identification accuracy improved.SOLUTION: A defect detection learning device 10 comprises: a pseudo defect image generation unit which generates pseudo defect images of an inspection target by using a non-defective product image of the inspection target; a determination network learning unit which learns a determination network 15k so as to perform a determination task of determining between the non-defective product image and the pseudo defect images; an adaptive network learning unit which learns an adaptive network 15m which performs prescribed feature quantity conversion so as to adapt a normal reference feature quantity of the non-defective product image extracted from the determination network 15k to a defect detection task; and a defect detection unit which detects defective products of the inspection target on the basis of the normal reference feature quantity after adaptation to the defect detection task. The pseudo defect images include images set as a target defect under a prescribed defect condition, and a ratio between the number of pseudo defect images of the target defect and the number of pseudo defect images of defects other than the target defect is adjusted through operation via the input unit 13.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to a fault detection learning device and the like. [Background technology]

[0002] As a technology for a computer to identify whether an object to be inspected is a good or bad product based on an image of the object to be inspected, for example, the technology described in Patent Document 1 is known. That is, Patent Document 1 describes an image inspection method including "a feature amount acquisition step of acquiring predetermined feature amounts in a transfer image and other objects to be inspected, and an automatic calculation step of automatically calculating a threshold value for newly classifying the object to be inspected into a good or bad product after measurement, based on the measurement results of the predetermined feature amounts obtained in the feature amount acquisition step." [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2023-51102 Summary of the Invention [Problem to be solved by the invention]

[0004] The technology described in Patent Document 1 creates a pseudo-image of a defective product by transferring the defective parts of an image that was previously identified as a defective product onto an image of a good product, but there is room for improvement in terms of improving the accuracy of identifying defective products, etc.

[0005] Therefore, an object of the present disclosure is to provide a defect detection learning device and the like that aims to improve identification accuracy. [Means for solving the problem]

[0006] In order to solve the above-mentioned problems, the defect detection learning device according to the present disclosure includes a pseudo-failure image generation unit that generates pseudo-failure images of an inspection object using a non-defective image of the inspection object; a discrimination network training unit that trains a discrimination network to perform a discrimination task of distinguishing between the non-defective image and the pseudo-failure image; an adaptive network training unit that trains an adaptive network that performs predetermined feature transformation so as to adapt normal reference features of the non-defective image extracted from the discrimination network to the defect detection task of the inspection object; and a defect detection unit that detects defects in the inspection object based on the normal reference features after adaptation to the defect detection task, wherein the pseudo-failure images include those that are set as target defects under predetermined defect conditions, and the ratio between the number of pseudo-failure images of the target defects and the number of other pseudo-failure images that are different from the target defects is adjusted by operation via an input unit. [Effects of the Invention]

[0007] According to the present disclosure, it is possible to provide a defect detection learning device and the like that aims to improve identification accuracy. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a functional block diagram including a failure detection learning device according to a first embodiment. [Figure 2] FIG. 2 is an explanatory diagram illustrating a hardware configuration of the defect detection learning device according to the first embodiment. [Figure 3] FIG. 2 is a functional block diagram of a processing unit of the failure detection learning device according to the first embodiment. [Figure 4] 4 is an example of a display screen for setting an inspection object, defect conditions, etc. in the defect detection learning device according to the first embodiment. [Figure 5] 3A to 3C are explanatory diagrams showing examples of non-defective product images and pseudo-defective images in the defect detection learning device according to the first embodiment. [Figure 6] 4 is a flowchart showing a flow of processing related to defect detection in the defect detection learning device according to the first embodiment. [Figure 7]FIG. 2 is an explanatory diagram of a classification network provided in the failure detection learning device according to the first embodiment. [Figure 8] FIG. 2 is an explanatory diagram of an adaptive network provided in the failure detection learning device according to the first embodiment. [Figure 9A] FIG. 2 is an explanatory diagram showing the relationship between the feature amounts of a non-defective image in a state where feature amount transformation has not been performed in the adaptive network and the normal reference feature amounts in the defect detection learning device according to the first embodiment. [Figure 9B] FIG. 2 is an explanatory diagram showing the relationship between feature amounts of a non-defective image after feature amount transformation in an adaptive network and normal reference feature amounts in the defect detection learning device according to the first embodiment. [Figure 10A] 4 is a flowchart related to generation of a pseudo-fault image in the fault detection learning device according to the first embodiment. [Figure 10B] 4 is a flowchart related to generation of a pseudo-fault image in the fault detection learning device according to the first embodiment. [Figure 11] 4 is a flowchart relating to learning of an adaptive network of the fault detection learning device according to the first embodiment. [Figure 12] 10 is a flowchart relating to learning of an adaptive network of the fault detection learning device according to the second embodiment. [Figure 13] 10 is a flowchart relating to learning of an adaptive network of the fault detection learning device according to the third embodiment. [Figure 14] 10 is a flowchart relating to a search for a fault condition in the fault detection learning device according to the fourth embodiment. [Figure 15] 13 is a flowchart related to calculation of integrated detection performance in the fault detection learning device according to the fifth embodiment. [Figure 16] 13 is a flowchart showing a flow of processing related to defect detection in the defect detection learning device according to the sixth embodiment. [Figure 17] FIG. 13 is a functional block diagram of an image inspection system according to a seventh embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] First Embodiment <Configuration of the defect detection learning device> FIG. 1 is a functional block diagram including a failure detection learning device 10 according to the first embodiment. The defect detection learning device 10 is a device for learning predetermined data used to detect defective products among inspection objects. Examples of types of inspection objects include, but are not limited to, industrial products, daily necessities, and food, as well as human and animal cells, blood vessels, and organs (X-ray images, etc.). In other words, anything that can be imaged by the imaging device 20 can be used as an inspection object.

[0010] A computer such as a personal computer, tablet, or smartphone may be used as such a defect detection learning device 10. Furthermore, the defect detection learning device 10 may be configured by connecting multiple computers in a predetermined manner via a communication line or a network. For example, the functions of the defect detection learning device 10 may be distributed among multiple computers such as a cloud server or an edge server. Note that detecting a defective product in an inspection object based on a captured image of the inspection object is referred to as "defect detection."

[0011] 1 is a device that captures an image of the surface or interior of an object to be inspected and generates the captured image as digital data. Examples of such an imaging device 20 include a CCD (Charge Coupled Device) camera, an optical microscope, a charged particle microscope, an ultrasonic inspection device, and an X-ray inspection device. The captured image of the object to be inspected generated by the imaging device 20 is transmitted to the defect detection learning device 10.

[0012] 1, the defect detection learning device 10 includes a processing unit 11, a communication unit 12, an input unit 13, a display unit 14, and a storage unit 15. The processing unit 11 reads various programs and data stored in the storage unit 15 and executes predetermined processing related to defect detection. As the processing unit 11, a processor such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or an FPGA (Field Programmable Gate Array) is used.

[0013] The communication unit 12 performs predetermined communication with an external device (not shown). For example, the communication unit 12 acquires captured images from the imaging device 20 via a LAN (Local Area Network) or Wi-Fi (registered trademark). The input unit 13 is a device for inputting predetermined instructions and selections based on input operations by the user. As such an input unit 13, for example, a keyboard, a mouse, a touch panel, a microphone, or the like may be used. The display unit 14 displays the calculation results of the processing unit 11 in a predetermined manner. For example, a display is used as the display unit 14. Note that a device that combines the functions of the input unit 13 and the display unit 14, such as a touch panel, may also be used.

[0014] Various programs and data are stored in the storage unit 15. As shown in Fig. 1, the storage unit 15 pre-stores, as predetermined programs, a pseudo-fault image generation program 15a, a discrimination network training program 15b, an adaptive network training program 15c, a fault detection program 15d, and a GUI execution program 15e.

[0015] The storage unit 15 also stores a pseudo-failure database 15f, a non-defective product database 15g, and an other information database 15h as predetermined databases. Each of these databases is generated by the processing unit 11. In addition to the above-mentioned data, the storage unit 15 also stores an identification network 15k, an adaptive network 15m, and an abnormality degree evaluation engine 15n. The identification network 15k and the adaptive network 15m are generated by the processing unit 11. The abnormality degree evaluation engine 15n is stored in advance as a predetermined program. Details of the data stored in the storage unit 15 will be described later.

[0016] FIG. 2 is an explanatory diagram showing the hardware configuration of the defect detection learning device 10. As shown in FIG. As shown in FIG. 2, the fault detection learning device 10 has, as its hardware configuration, a processor 111, a RAM 151 (Random Access Memory), a ROM 152 (Read Only Memory), a HDD 153 (Hard Disk Drive), a communication interface 16, an input / output interface 17, and a media interface 18, which are connected in a predetermined manner via a bus 19. The processor 111 shown in FIG. 2 is the hardware of the processing unit 11 (see FIG. 1). The RAM 151, ROM 152, and HDD 153 are the hardware of the storage unit 15 (see FIG. 1).

[0017] The processor 111 reads out a predetermined program stored in the ROM 152 or the HDD 153 and loads it into the RAM 151, thereby executing a predetermined process. As shown in Fig. 2, the communication unit 12 is connected to a bus 19 via a communication interface 16. The input unit 13 and the display unit 14 are connected to the bus 19 via an input / output interface 17. A recording medium 30 is appropriately connected to the media interface 18. Information is read from the recording medium 30 and information is written to the recording medium 30 via the media interface 18. Note that the hardware configuration shown in Fig. 2 is an example and is not limited to this.

[0018] Returning to Figure 1 again, the explanation will continue. The pseudo-failure image generating program 15a shown in Fig. 1 is a program for generating a pseudo-failure image that shows a pseudo-image of a defective product under inspection. Here, a "pseudo-failure image" is an image in which a pseudo-prescribed image pattern is synthesized with a part of a good-product image of the inspection target (see Fig. 5). Also, a "good-product image" is an image of an inspection target that is known to be a good product. The pseudo-failure image generating program 15a stores data of the pseudo-failure image in a pseudo-failure database 15f.

[0019] The classification network learning program 15b is a program for performing machine learning of the classification network 15k. The adaptive network learning program 15c is a program for performing machine learning of the adaptive network 15m. Details of the classification network 15k and the adaptive network 15m will be described later.

[0020] The defect detection program 15d extracts feature quantities from the non-defective images and the pseudo-defective images used for defect detection. The defect detection program 15d also has a function to evaluate whether the defect detection satisfies a predetermined detection performance. The GUI execution program 15e accepts the designation of the inspection object and the defect conditions based on the input operation of the user via the GUI (Graphical User Interface), and also accepts the input of data such as the threshold value of the detection performance. The GUI execution program 15e also displays the pseudo-defective image, the non-defective image, and the defect detection performance data on the display unit 14.

[0021] The pseudo-fault database 15f is a database in which a group of pseudo-fault images generated by the pseudo-fault image generating program 15a is stored. Note that the feature quantities of the pseudo-fault images generated using the classification network 15k are also stored in the pseudo-fault database 15f.

[0022] The non-defective product database 15g is a database that stores a group of non-defective product images obtained by imaging with the imaging device 20. The non-defective product database 15g also stores normal reference features generated from the group of non-defective product images using the classification network 15k. Details of the normal reference features will be described later.

[0023] The other information database 15h stores various information used in each defect detection process. For example, the other information database 15h stores data on defect detection performance as well as inspection objects and defect conditions specified by input operations via the input unit 13.

[0024] The classification network 15k is a neural network for performing a predetermined classification task on non-defective images and pseudo-defective images, and is generated by the classification network training program 15b. Here, the "classification task" refers to a process of distinguishing between non-defective images and pseudo-defective images, or identifying the type of defect in a pseudo-defective image. Such classification tasks include image classification and semantic segmentation (area classification).

[0025] The adaptive network 15m is a neural network for adapting normal reference feature amounts of non-defective product images to the task of detecting defects in an inspection target, and is generated by the adaptive network learning program 15c. The abnormality degree evaluation engine 15n calculates the degree of abnormality in the target image by the defect detection program 15d, and if the degree of abnormality is equal to or greater than a predetermined threshold, judges (evaluates) the image as a defective image.

[0026] FIG. 3 is a functional block diagram of the processing unit 11 of the fault detection learning device. As shown in FIG. 3, the processing unit 11 has, as its functional configuration, a pseudo-fault image generating unit 11a, a discrimination network learning unit 11b, an adaptive network learning unit 11c, a fault detecting unit 11d, and a display control unit 11e.

[0027] The pseudo-failure image generating unit 11a generates a pseudo-failure image of the object to be inspected by using a non-defective image of the object to be inspected. Such processing is performed by the processing unit 11 executing the pseudo-failure image generating program 15a (see FIG. 1). The classification network training unit 11b trains the classification network 15k (see FIG. 1) to perform a classification task of distinguishing between non-defective images and pseudo-defective images. Such processing is performed by the processing unit 11 executing the classification network training program 15b (see FIG. 1).

[0028] The adaptive network training unit 11c trains an adaptive network 15m (see FIG. 1) that performs predetermined feature transformation so as to adapt the normal reference feature of a non-defective product image extracted from the classification network 15k (see FIG. 1) to the task of detecting defects in an inspection object. Such processing is performed by the processing unit 11 executing the adaptive network training program 15c (see FIG. 1).

[0029] The defect detection unit 11d extracts feature amounts from the images of good products and pseudo-failure images using the defect detection program 15d (see FIG. 1), and also determines whether the defect detection satisfies a predetermined detection performance. The defect detection unit 11d also has a function of detecting defective products using the captured image of the inspection object based on the normal reference feature amounts after adaptation to the defect detection task. The display control unit 11e causes a GUI execution program 15e (see FIG. 1) to display predetermined processing results on the display unit 14 (see FIG. 1). Details of each component shown in FIG. 3 will be described later.

[0030] It is known that when detecting defects in an inspection object using deep learning, using both images of good and bad products to train a neural network increases the accuracy of detecting defective products. However, because the occurrence of defective products is minimized on the production line, it is difficult to collect a sufficient number of images of defects. Even if a sufficient number of images of defects could be collected, it would require a great deal of time and effort to associate each image with its type (annotation). Therefore, in the first embodiment, the processing unit 11 generates pseudo-defective images using images of good products. Next, various settings by the user when performing machine learning for defect detection (corresponding to step S101 in the flowchart of FIG. 6) will be described with reference to FIG. 4.

[0031] FIG. 4 shows an example of a display screen for setting the inspection object, defect conditions, etc. The parameter setting field F1 on the display screen shown in Fig. 4 includes a setting field F11 for the object to be inspected, as well as multiple setting fields F12 to F16 for target defects. Here, "target defects" refer to the type of defect that is targeted when improving the detection accuracy in defect detection of the object to be inspected. Generally, there are multiple types of defects in the object to be inspected, but the type of defect for which the user wishes to particularly improve the detection accuracy, taking into account past inspection results, etc., is a "target defect."

[0032] In the inspection object setting field F11 shown in FIG. 4, the type and model of the inspection object are set based on user operation via the input unit 13 (see FIG. 1). In the target defect position setting field F12, the position of the target defect in the entire image is set. In the target defect size setting field F13, the size (number of pixels) of the target defect is set. In the target defect shape setting field F14, the shape of the target defect is set. In the target defect image pattern number setting field F15, the total number of types of target defect image patterns is set. Details of the target defect image patterns will be described later. In the detection performance threshold setting field F16, a threshold (e.g., 95%) for the defect detection rate (correct rate) when detecting target defects is set. Although omitted in FIG. 4, setting conditions for defects other than target defects related to the inspection object are also entered as appropriate.

[0033] 4, the ID (identification information) of a non-defective image is selected from a pull-down list by operating the input unit 13 (see FIG. 1). When a predetermined non-defective image ID is selected, the corresponding non-defective image is displayed in the image display area F3.

[0034] In the pseudo-failure image ID field F4 for target failure, the ID (identification information) of the pseudo-failure image for the target failure is selected from a pull-down list by operation via the input unit 13 (see FIG. 1). When a specific pseudo-failure image ID is selected, the corresponding pseudo-failure image is displayed in the image display area F5. This allows the user to confirm the pseudo-failure image for the target failure.

[0035] The graph display area F6 displays the transition of the detection performance of the adaptive network 15m (see FIG. 1). The horizontal axis of the graph displayed in the graph display area F6 represents the number of learning times (the number of epochs), and the vertical axis represents the defect detection rate. The "defect detection rate" mentioned above is the accuracy rate indicating the degree to which pseudo-defective images were correctly detected as "defective." As shown in FIG. 4, by displaying the transition of the defect detection rate as a graph, the user can grasp the progress of the learning. In this way, the display control unit 11e (see FIG. 3) displays the pseudo-defective images of the target defects on the display unit 14 (see FIG. 2), and also displays the transition of the defect detection rate, which indicates the detection performance of the target defects, on the display unit (see FIG. 2). Next, specific examples of non-defective images and pseudo-defective images will be described with reference to FIG.

[0036] FIG. 5 is an explanatory diagram showing examples of a non-defective image G1 and pseudo-defective images G2 to G10. The good-quality image G1 in the upper left of FIG. 5 is generated by capturing an image of the inspection object 1, which is known to be a good product, using the imaging device 20 (see FIG. 1). The remaining pseudo-failure images G2 to G10 are generated by combining predetermined pseudo-failures with a portion of the good-quality image G1. In the example of FIG. 5, the inspection object 1 is configured to include three portions 1a, 1b, and 1c. As variations of such a defective product of the inspection object 1, for example, three variations of the pseudo-failure images (referred to as "failure variations") can be given from the viewpoint of providing a pseudo-failure in one of the portions 1a, 1b, and 1c.

[0037] Furthermore, from the viewpoint of the size of the defect portion, each of the three cases can be divided into cases where the pseudo-defect size is large and cases where the pseudo-defect size is small. For example, pseudo-defect images G2 and G5 show a case where a pseudo-defect is present in portion 1b of the inspection object 1. More specifically, pseudo-defect image G2 has a large pseudo-defect D2 in portion 1b, and pseudo-defect image G5 has a small pseudo-defect D5 in portion 1b. Similarly, pseudo-defect images G3 and G6 show a case where a pseudo-defect of a predetermined size is present in portion 1a of the inspection object 1. Furthermore, pseudo-defect images G4 and G7 show a case where a pseudo-defect of a predetermined size is present in portion 1c of the inspection object 1.

[0038] For example, assume that past image inspection results show that small defects present in portion 1b are particularly likely to be overlooked. In such a case, in order to improve the accuracy of detecting defects in portion 1b, processing unit 11 (see FIG. 1) generates pseudo-defects with various image patterns and combines the pseudo-defects with part of good product image G1. For example, processing unit 11 (see FIG. 1) may paste the pseudo-defect image pattern directly onto the good product image. Alternatively, the image may be processed in a predetermined manner so that the boundary between the pseudo-defect image pattern and the good product image is seamless.

[0039] In the example of FIG. 5, three pseudo-failure images G8, G9, and G10 are generated as failure variations when the size of the pseudo-failure in portion 1b is small. As shown in the enlarged partial views of the pseudo-failures D8, D9, and D10 at the bottom of FIG. 5, these pseudo-failures D8, D9, and D10 all have different image patterns. Specifically, the pseudo-failure D8 has an image pattern including multiple horizontal line segments. The pseudo-failure D9 has an image pattern including multiple vertical line segments. The pseudo-failure D10 has an image pattern including multiple diagonal line segments. The image patterns of the target failures and the number of failure variations can be changed as appropriate.

[0040] Furthermore, the pseudo-failure image generating unit 11a (see FIG. 3) generates pseudo-failure images of multiple types of target defects (three types in the example of FIG. 5) for each image pattern. Here, it is preferable that the contour shapes of the image patterns are common in the pseudo-failure images of multiple types of target defects. In the example of FIG. 5, the contours of the image patterns of pseudo-failures D8, D9, and D10 are all circular. By making the contours of multiple types of image patterns the same shape in this way, the processing unit 11 (see FIG. 1) can identify differences in image patterns with high accuracy when detecting defects.

[0041] The good product image G1 and the pseudo-fault images G2 to G10 shown in FIG. 5 are used for training a classification network 15k (see FIG. 1) and an adaptive network 15m (see FIG. 1), which will be described later, and for evaluation after training. Note that, in practice, hundreds, thousands, or tens of thousands of good product images and pseudo-fault images are used during training and evaluation. In this case, the good product images include a plurality of good product images for training and a plurality of good product images for evaluation. Similarly, the pseudo-fault images include a plurality of pseudo-fault images for training and a plurality of pseudo-fault images for evaluation.

[0042] FIG. 6 is a flowchart showing the flow of processing related to defect detection (also refer to FIG. 1 as appropriate). In step S101, the processing unit 11 of the defect detection learning device 10 sets the inspection object and the defect conditions. That is, the processing unit 11 stores the data of the inspection object and the defect conditions input by operation via the input unit 13 in the other information database 15h. The result of the processing in step S101 is displayed as a predetermined display screen as shown in FIG.

[0043] For example, as shown in FIG. 5, pseudo defects D5, D8, D9, and D10, small-sized defects in portion 1b of the inspection object 1 may be set as "target defects." In this case, portion 1b of the inspection object 1 (see FIG. 5) is specified in field F12 for setting the position of the "target defect" shown in FIG. 4. Also, in field F13 for setting the size, "smaller size" (or the number of pixels in the defective portion) is selected. In field F14 for setting the shape, "circle" is selected. In field F15 for setting the number of image patterns, "3" is selected, which is the number of types of pseudo defects D8, D9, and D10 (see FIG. 5).

[0044] Next, in step S102 of Fig. 6, the processing unit 11 generates pseudo-failure images (pseudo-failure image generation step). That is, the processing unit 11 reads input information on the inspection object and the defect conditions from the other information database 15h and also reads the non-defective product images from the non-defective product database 15g by the pseudo-failure image generation program 15a. Then, the defect detection learning device 10 generates a group of pseudo-failure images based on the input information on the inspection object and the defect conditions and the non-defective product images. The method for generating the pseudo-failure image group is as described above. For example, pseudo-failure images G2 to G10 are generated from the non-defective product image G1 shown in Fig. 5.

[0045] As described above, the pseudo-failure images include those set as target failures under predetermined failure conditions. The failure conditions may be set by an operation via the input unit 13, or the predetermined failure conditions may be extracted from the storage unit 15 that stores past failure conditions of the object to be inspected. In this case, it is preferable that the number (total number) of pseudo-failure images of target failures is greater than the number of each type of pseudo-failure images different from the target failures.

[0046] For example, if 1000 pseudo-fault images each of G8, G9, and G10 shown in Figure 5 are generated, the total number of pseudo-fault images for the target defects will be 3000. Also, suppose that there are 1000 pseudo-fault images for each of the cases where there are pseudo-faults (including large and small sizes) in portion 1a of the object 1 to be inspected, where there are large pseudo-faults in portion 1b, and where there are pseudo-faults (including large and small sizes) in portion 1c. In other words, suppose there are 1000 pseudo-fault images with pseudo-faults in portion a, 1000 pseudo-fault images with pseudo-faults in portion b, and 1000 pseudo-fault images with pseudo-faults in portion c.

[0047] In this case, the total number of pseudo-failure images corresponding to the target failure (3,000 images) is greater than the number of each type of other pseudo-failure images (pseudo-failure images other than the target failure) (1,000 images each). This allows the processing unit 11 to train the classification network 15k and the adaptive network 15m, thereby generating predetermined features with high classification performance for the target failure.

[0048] Although not shown in the display example of Fig. 4, the ratio between the number (total number) of pseudo-failure images of target failures and the number of other pseudo-failure images different from target failures can be adjusted by an operation via the input unit 13. This increases the degree of freedom in setting by the user. In addition, the user can appropriately set the number of pseudo-failure images of target failures to be greater than the number of each type of pseudo-failure images other than target failures.

[0049] 6, the processing unit 11 trains the classification network 15k (classification network training step). That is, the processing unit 11 reads a group of good-quality images for training from the good-quality database 15g and a group of pseudo-faulty images for training from the pseudo-faulty image database 14f using the classification network training program 15b. Then, the processing unit 11 trains the classification network 15k based on the group of good-quality images and the group of pseudo-faulty images for training.

[0050] FIG. 7 is an explanatory diagram of the identification network 15k. As described above, the classification network 15k shown in FIG. 7 is a neural network that is trained to distinguish between non-defective images and pseudo-defective images. As shown in FIG. 7, the classification network 15k includes an input layer, an intermediate layer, and an output layer. Image information such as non-defective images and pseudo-defective images is input to the input layer. For example, the image may be divided into multiple patch areas A1, and the value of the central pixel of the patch area A1 (a pixel value in which the values ​​of each pixel of the patch area A1 are reflected in a predetermined manner) may be input to a node in the input layer. Alternatively, the value of each pixel of the image (a pixel value in which the values ​​of its surrounding pixels are reflected in a predetermined manner) may be input to a node in the input layer.

[0051] Each node in the input layer is connected to each node in the first layer of the hidden layer. The hidden layer includes multiple layers and is located between the input layer and the output layer. A predetermined weight value is assigned to the branches connecting the nodes. Each node is assigned a predetermined activation function, which includes the sum of the values ​​of the multiple nodes connected upstream of that node multiplied by the weight. If the value of the activation function exceeds a limit value, that node outputs a predetermined value to the downstream node via the branch. The last layer of the hidden layer is connected to a node in the output layer. The combination of node values ​​in the output layer indicates a predetermined identification result for fault detection.

[0052] Note that the classification task using the classification network 15k may involve classifying images. For example, a method of classifying images may be used in which non-defective images are classified into one class and defective variations of pseudo-defective images are classified into one class, and the image class is estimated. Also, FIG. 7 is an example, and the classification network 15k may be, for example, a convolutional neural network.

[0053] 6, the processing unit 11 generates normal reference features. That is, the processing unit 11 reads the classification network 15k and also reads a group of good-quality product images for training from the good-quality product database 15g using the adaptive network training program 15c. Then, the processing unit 11 generates normal reference features based on the classification network 15k and the group of good-quality product images for training. Here, the "normal reference features" are predetermined features that serve as a reference for good-quality product images.

[0054] To explain the process of step S104 in more detail, the processing unit 11 inputs information about a non-defective image to the input layer of the classification network 15k (see FIG. 7). Then, the processing unit 11 extracts a combination of values ​​of each node in a predetermined layer Q1 included in the intermediate layer of the classification network 15k as a normal reference feature. The combination of values ​​of each node in the predetermined layer Q1 can be regarded as a feature (position vector) indicating a predetermined point in a feature space (multidimensional vector space). The layer Q1 from which the normal reference feature is extracted is preset in the adaptive network training program 15c.

[0055] Incidentally, the number of features of non-defective images tends to increase depending on the number of non-defective images and the number of patch areas A1 set. Therefore, the processing unit 11 may determine normal reference features after appropriately merging or thinning out the features so that the number of features obtained from the group of non-defective images for learning falls within a predetermined range. The multiple normal reference features generated in this manner are stored in the non-defective product database 15g.

[0056] 6, the processing unit 11 trains the adaptive network 15m (adaptive network training step). That is, the processing unit 11 reads normal reference features and a group of good-quality images for training from the good-quality product database 15g using the adaptive network training program 15c. Then, the processing unit 11 trains the adaptive network 15m based on the normal reference features and the group of good-quality product images for training.

[0057] FIG. 8 is an explanatory diagram of the adaptive network 15m. As described above, the adaptive network 15m is a neural network for adapting normal reference features of non-defective images to the task of detecting defects in an object to be inspected. Similar to the classification network 15k (see FIG. 7), the adaptive network 15m includes an input layer, multiple intermediate layers, and an output layer. A convolutional neural network may be used as the adaptive network 15m.

[0058] The processing unit 11 first inputs data of a non-defective image into the classification network 15k (see FIG. 7) and extracts features of the non-defective image from a predetermined layer Q1 (see FIG. 7). The processing unit 11 inputs the features of the non-defective image extracted from the classification network 15k into the adaptive network 15m. The processing unit 11 then obtains, as the output of the adaptive network 15m, features adapted to the task of detecting defects in the inspection object. Note that the feature conversion using the adaptive network 15m is performed on normal reference features as well as the features of the non-defective image described above.

[0059] FIG. 9A is an explanatory diagram showing the relationship between feature amounts U1, U2, and U3 of a non-defective image in a state where feature amount transformation by the adaptive network has not been performed, and normal reference feature amounts V1, V2, and V3. The multiple normal reference features V1, V2, and V3 shown in Figure 9A are features to be converted by the adaptive network 15m (see Figure 8). The multiple non-defective image features U1, U2, and U3 shown in Figure 9A are features extracted from layer Q1 (see Figure 7) of the classification network 15k (see Figure 7) using non-defective images as input. The range C1 indicated by a circular dashed line in Figure 9A is the range of non-defective image features centered on the normal reference feature V1. The same applies to the other ranges C2 and C3 indicated by circular dashed lines.

[0060] Before feature conversion using adaptive network 15m (see FIG. 8), for example, some of the multiple feature quantities U1 of a non-defective image are slightly outside the range C1 based on the normal reference feature quantity V1. In other words, among the multiple feature quantities U1 of a non-defective image, there are some feature quantities that are difficult to distinguish as being a non-defective product (within the range C1) or a defective product (outside the range C1). The same is true for the other feature quantities U2 and U3 of the non-defective image.

[0061] FIG. 9B is an explanatory diagram showing the relationship between feature amounts U1, U2, and U3 of a non-defective image after feature amount transformation by the adaptive network and normal reference feature amounts V1, V2, and V3. As shown in Fig. 9B, the adaptive network 15m (see Fig. 8) is trained so that the feature U1 of the non-defective image is included in the range C1 based on the normal reference feature V1. The same applies to the other feature U2 and U3 of the non-defective image.

[0062] This improves the discrimination performance between good-product images and pseudo-failure images (especially target defects). When projecting the feature values ​​of the good-product images near the normal reference feature values ​​in this way, a predetermined convolutional neural network may be used as the adaptive network 15m. The details of the learning of the adaptive network 15m (see FIG. 8) will be described later.

[0063] 6, the processing unit 11 evaluates the detection performance of pseudo-fault images. That is, the processing unit 11 reads a group of pseudo-fault images for evaluation from the pseudo-fault database 15f, and also reads normal reference features and a group of good-product images for evaluation from the good-product database 15g, using the defect detection program 15d. Then, the processing unit 11 generates features for the group of good-product images and also generates features for the group of pseudo-fault images, using the classification network 15k.

[0064] Furthermore, the processing unit 11 uses the adaptive network 15m to convert the normal reference feature, the feature of the non-defective image group, and the feature of the pseudo-fault image group. If the distance from the normal reference feature to the feature of the pseudo-fault image is equal to or greater than a predetermined threshold, the processing unit 11 determines that a defect exists in the pseudo-fault image. Since it is known that a defect exists in the pseudo-fault image, the identification result in this case (defect exists) can be said to be correct. On the other hand, if the distance from the normal reference feature is less than the predetermined threshold, the processing unit 11 determines that no defect exists in the pseudo-fault image. The identification result in this case (no defect exists) is incorrect. In this way, defect detection is performed for each of the numerous pseudo-fault image groups to evaluate the identification accuracy based on machine learning.

[0065] In step S107, the processing unit 11 loads the abnormality evaluation engine 15n using the defect detection program 15d, and determines whether the detection performance (defect detection rate) of the pseudo-defective images is equal to or greater than a predetermined threshold. If the detection performance of the pseudo-defective images is less than the predetermined threshold (step S107: No), the processing unit 11 returns to step S105. Then, in step S105, the processing unit 11 performs training of the adaptive network 15m again (i.e., performs training for the next epoch). Note that the index of detection performance may be set so that the defect detection rate for the target defect is ≥ 95% and the variation in the defect detection rate between training sessions is ≤ 1%.

[0066] Furthermore, if the detection performance of the pseudo-fault image is equal to or greater than the predetermined threshold in step S107 (step S107: Yes), the processing unit 11 proceeds to step S108. In step S108, the processing unit 11 saves the parameters of the trained adaptive network 15m. In this way, the processing unit 11 determines whether to terminate the training of the adaptive network 15m based on the evaluation result of the detection performance of the pseudo-fault image including the target fault. That is, the adaptive network training unit 11c performs fault detection on the pseudo-fault image of the target fault, and terminates the training of the adaptive network 15m based on the evaluation of the detection performance in the fault detection. For example, the adaptive network training unit 11c terminates the training of the adaptive network 15m when the detection performance (fault detection rate) indicating the detection performance on the pseudo-fault image of the target fault becomes equal to or greater than the predetermined threshold.

[0067] This makes it possible to stably generate adaptive network 15m (see FIG. 8) with high detection performance for target defects. Although not shown in FIG. 6, the transition of detection performance (fault detection rate) may be displayed on display unit 14 (see graph display area F6 in FIG. 4).

[0068] Next, in step S109 of FIG. 6, the processing unit 11 detects defects in the test image. That is, the processing unit 11 performs defect detection on a predetermined test image using the defect detection program 14d, the classification network 15k, the adaptive network 15m, the anomaly evaluation engine 15n, and the normal reference feature. Then, although not shown in FIG. 6, if a defect in the test image can be properly detected in the processing of step S109, the processing unit 11 ends the series of processes (END). Note that, although not shown in FIG. 6, the defect detection unit 11d detects defective products using a captured image of the inspection object based on the normal reference feature after adaptation to the defect detection task (defect detection step). This enables detection of defective products with high accuracy.

[0069] 10A and 10B are flowcharts relating to the generation of pseudo-fault images (see also FIG. 1 as appropriate). 10A and 10B correspond to step S102 (generation of pseudo-fault images) in Fig. 6. For simplicity, let K be the number of pseudo-fault images included in each class of pseudo-fault images (including target defects) in machine learning. Steps S102a to S102j in FIG. 10A are processes related to the generation of pseudo-failure images other than the target failure (for example, pseudo-failure images G2 to G4, G6, and G7 in FIG. 5).

[0070] In step S102a, the processing unit 11 reads the defective conditions, etc. That is, the processing unit 11 reads input information on the inspection target and the defective conditions from the other information database 15h by the pseudo-failure image generation program 15a, and also reads a group of non-defective product images from the non-defective product database 15g.

[0071] In step S102b, the processing unit 11 generates an image pattern of a pseudo-failure using the pseudo-failure image generation program 15a. For example, the processing unit 11 may generate an image pattern of a pseudo-failure based on a predetermined mathematical formula. A class is assigned to the image pattern by linking the parameters of this mathematical formula to the identification information of the class. Note that, for image patterns of defect variations other than target defects, there is no particular need to classify them into multiple classes, so they may be treated as a single class. Alternatively, an image pattern may be generated based on a predetermined noise.

[0072] In step S102c, the processing unit 11 repeats the processes of steps S102d to S102i until the value m starts from 1 and reaches the value M, which is the number of classes other than the target failure. As a result, a group of pseudo failure images of a plurality of failure variations other than the target failure is generated.

[0073] In step S102d, the processing unit 11 repeats the processes of steps S102e to S102h, starting from 1, until the value k reaches the value K, which is the number of pseudo-fault images for each class. This generates a group of pseudo-fault images of predetermined fault variations (i.e., classes) other than the target fault. In step S102e, the processing unit 11 reads a predetermined non-defective image from the group of non-defective images by the pseudo-defective image generating program 15a.

[0074] In step S102f, the processing unit 11 reads an image pattern corresponding to a predetermined defect variation other than the target defect by using the pseudo defect image generating program 15a. In step S102g, the processing unit 11 cuts out an image pattern having a shape and size corresponding to a predetermined defect variation other than the target defect from the image pattern (the image pattern read in step S102f) by the pseudo defect image generating program 15a.

[0075] In step S102h, the processing unit 11 synthesizes an image pattern at a predetermined position on the good product image by using the pseudo-failure image generation program 15a. That is, the processing unit 11 synthesizes an image pattern at a position on the good product image that corresponds to a predetermined failure variation other than the target failure, thereby generating a pseudo-failure image. The position of the image pattern to be synthesized on the good product image may be set randomly so as to be different from that in the case of the target failure.

[0076] In step S102i, if the value k has reached the value K, which is the number of pseudo-fault images in each class, the processing unit 11 ends the loop of step S102d. If the value k has not reached the value K, the processing unit 11 repeats the processes (steps S102e to S102h) related to the loop of step S102d.

[0077] In step S102j, if the value m has reached the value M, which is the number of classes other than the target failure, the processing unit 11 ends the loop of step S102c. On the other hand, if the value m has not reached the value M, the processing unit 11 repeats the processing related to the loop of step S102c (steps S102d to S102i). In this way, a group of pseudo-failure images other than the target failure is generated. Next, the processing of the processing unit 11 proceeds to step S102k in FIG. 10B.

[0078] Steps S102k, S102m, S102n, and S102p to S102t in FIG. 10B are processes related to generating a group of pseudo-failure images of target failures (for example, pseudo-failure images G5, G8 to G10 in FIG. 5). In step S102k, the processing unit 11 repeats the processes of steps S102m, S102n, and S102p to S102s until the value n starts from 1 and reaches the value N, which is the number of target defect classes. As a result, a group of pseudo defect images of multiple defect variations of the target defect is generated.

[0079] In step S102m, the processing unit 11 repeats the processes of steps S102n and S102p to S102r until the value k starts from 1 and reaches the value K, which is the number of pseudo-fault images for each class. In this way, a group of pseudo-fault images of a predetermined fault variation (i.e., class) of the target fault is generated. In step S102n, the processing unit 11 reads a predetermined non-defective image from the group of non-defective images by the pseudo-defective image generating program 15a.

[0080] In step S102p, the processing unit 11 reads an image pattern corresponding to a predetermined failure variation of the target failure by the pseudo failure image generating program 15a. In step S102q, the processing unit 11 uses the pseudo-failure image generation program 15a to extract image patterns of a shape and size common to the classes from the image patterns (image patterns read in step S102p). For example, the processing unit 11 extracts image patterns such as pseudo-failures D8, D9, and D10 shown in FIG. 5. In this way, the shape and size of the image patterns of pseudo-failures are made common between target failure classes. This makes it possible to generate features with high discrimination performance, particularly focusing on differences in image patterns.

[0081] In step S102r, the processing unit 11 synthesizes an image pattern at a predetermined position of the good product image by using the pseudo-fault image generation program 15a. That is, the processing unit 11 synthesizes an image pattern at a position corresponding to the target fault in the good product image, thereby generating a pseudo-fault image. In the example of Fig. 5, the pseudo-faults D8, D9, and D10 are located at the same position, but the positions of the pseudo-fault image patterns may be set randomly.

[0082] In step S102s, if the value k has reached the value K, which is the number of pseudo-fault images for each class, the processing unit 11 ends the loop of step S102m. If the value k has not reached the value K, the processing unit 11 repeats the processes (steps S102n, S102p to S102r) related to the loop of step S102m.

[0083] In step S102t, if the value n has reached the value N, which is the number of target defect classes, the processing unit 11 ends the loop of step S102k. On the other hand, if the value n has not reached the value N, the processing unit 11 repeats the processing related to the loop of step S102k (steps S102m, S102n, S102p to S102s). In this way, a group of pseudo defect images of target defects is generated.

[0084] FIG. 11 is a flowchart for training an adaptive network (see also FIG. 1 where appropriate). The series of processes shown in FIG. 11 corresponds to step S105 (learning of the adaptive network 15m) in FIG. In step S105a, the processing unit 11 generates feature quantities for the non-defective product images. First, the processing unit 11 reads the classification network 15k using the adaptive network learning program 15c, and also reads a group of non-defective product images for learning from the non-defective product database 15g, to generate feature quantities for the non-defective product images. Furthermore, the processing unit 11 converts the feature quantities of the non-defective product images using the adaptive network 15m. Note that the methods of extracting feature quantities based on the classification network 15k (see FIG. 7) and converting feature quantities based on the adaptive network 15m (see FIG. 8) are as described above.

[0085] In step S105b, the processing unit 11 calculates the distance between the feature amounts of the non-defective images. Specifically, the processing unit 11 calculates the distance between the feature amount of the non-defective image and the normal reference feature amount in the feature amount space (the distance between the feature amounts of the non-defective images) using the adaptive network learning program 15c. As a method for calculating this distance, for example, a nearest neighbor search method is used. For example, the processing unit 11 determines the feature amount U1 of the multiple non-defective images shown in FIG. 9B with the shortest distance as the calculation result of step S105b.

[0086] In step S105c, the processing unit 11 calculates the distance loss between the feature quantities of the non-defective images. That is, the processing unit 11 calculates the distance loss based on the distance between the feature quantities of the non-defective images using the adaptive network learning program 15c. Note that a formula for calculating the distance loss is preset so that the value increases as the distance between the feature quantities of the non-defective images after conversion by the adaptive network 15m deviates from a predetermined range.

[0087] In step S105d, the processing unit 11 generates an anomaly degree map of the non-defective image. That is, the processing unit 11 reads the anomaly degree evaluation engine 15n using the adaptive network learning program 15c, and calculates the local anomaly degree (for each patch region or each pixel) of the non-defective image. For example, the processing unit 11 clusters each feature amount of the normal reference feature amount group, and calculates this anomaly degree based on the average value or maximum value of the distance between the representative point (cluster center) of each class and the feature amount belonging to that class. That is, the larger the average value or maximum value of the distance, the higher the anomaly degree. Then, the processing unit 11 generates an anomaly degree map in which an anomaly degree value is associated with each patch region or each pixel of the non-defective image.

[0088] In step S105e, the processing unit 11 calculates the anomaly loss based on the anomaly map using the adaptive network learning program 15c. Note that a formula for calculating the anomaly loss is preset so that the value increases as the value of the anomaly map for non-defective images (for example, the sum of the anomalies) increases. In step S105f, the processing unit 11 updates the parameters of the adaptive network 15m based on the losses. That is, the defect detection learning device 10 updates the parameters of the adaptive network 15m (such as the weight of each branch) using the adaptive network learning program 15c so that the combined loss value of the distance loss and the anomaly loss for non-defective images becomes smaller. By using the distance loss and the anomaly loss in this way for learning the adaptive network 15m, it becomes possible to convert them into features adapted to the defect detection task.

[0089] The "loss" obtained by combining the distance loss and the anomaly loss may be calculated as the sum of the distance loss and the anomaly loss, or may be calculated as the sum of values ​​obtained by multiplying the distance loss or the anomaly loss by a predetermined coefficient. The adaptive network training unit 11c (see FIG. 3) may train the adaptive network 15m so that the anomaly level of non-defective images is equal to or less than a predetermined threshold. This allows the adaptive network 15m to be trained to perform feature transformation adapted to the defect detection task.

[0090] <Effects> According to the first embodiment, the adaptive network 15m performs a predetermined feature transformation so as to adapt the normal reference feature of a non-defective product image extracted from the classification network 15k to a defect detection task for an inspection target, thereby improving classification accuracy when detecting defects. In addition, the number of pseudo-failure images of the target failure is set to be greater than the number of each type of pseudo-failure image different from the target failure. This allows the classification network 15k and the adaptive network 15m to learn based on target failures with various failure variations, thereby improving the classification performance, especially for target failures.

[0091] Furthermore, according to the first embodiment, whether or not to terminate learning of the adaptive network 15m is determined based on the evaluation of the detection performance of the pseudo-fault images, which makes it possible to stably generate features with high discrimination performance for pseudo-fault images (especially target defects). Furthermore, there is no particular need to use captured images of actual defective products; pseudo-defective images are generated by processing images of good products. As mentioned above, since the occurrence of defective products is minimized on the production line, it is difficult to collect a sufficient number of defective images. In contrast, in the first embodiment, there is no particular need to use captured images of defective products, so a practical defect detection learning device 10 can be provided.

[0092] Second Embodiment The second embodiment differs from the first embodiment in the way adaptive network 15m (see FIG. 8) learns, but other aspects (such as the configuration of fault detection learning device 10: see FIG. 1) are the same as those of the first embodiment. Therefore, only the parts that differ from the first embodiment will be described, and explanations of overlapping parts will be omitted.

[0093] FIG. 12 is a flowchart relating to learning of the adaptive network of the fault detection learning device according to the second embodiment (also see FIG. 1 as appropriate). The series of processes shown in Fig. 12 corresponds to step S105 (learning of the adaptive network 15m) in Fig. 6. Furthermore, the processes in steps S105a to S105c in Fig. 12 are similar to steps S105a to S105c in the first embodiment (see Fig. 11), and therefore description thereof will be omitted.

[0094] After calculating the distance loss between the feature quantities of the non-defective images in step S105c of Fig. 12, the processing unit 11 proceeds to step S105g. In step S105g, the processing unit 11 generates feature quantities of the pseudo-defective images. That is, the processing unit 11 reads the classification network 15k by the adaptive network learning program 15c, and also reads a group of pseudo-defective images for learning from the pseudo-defective database 15f, and generates feature quantities of the pseudo-defective images. The method for generating feature quantities of the pseudo-defective images is the same as that for the non-defective images.

[0095] In step S105h, the processing unit 11 calculates the distance between the feature amounts of the non-defective image and the pseudo-fault image. That is, the processing unit 11 calculates the distance between the normal reference feature amount and the feature amount of the pseudo-fault image in the feature amount space by the adaptive network learning program 15c. Note that there are multiple normal reference feature amounts and multiple feature amounts of the pseudo-fault image, and the maximum, minimum, or average value of the distance between the two may be used as appropriate.

[0096] In step S105i, the processing unit 11 calculates the distance loss between the feature amounts of the good-product image and the pseudo-defective image. That is, the processing unit 11 calculates the distance loss between the feature amounts of the good-product image and the pseudo-defective image based on the distance between the feature amounts of the good-product image and the pseudo-defective image using the adaptive network learning program 15c. Note that a formula for calculating the distance loss is preset so that the value increases as the distance between the feature amount of the pseudo-defective image converted by the adaptive network 15m and the feature amount of the good-product image decreases.

[0097] In step S105j, the processing unit 11 updates the parameters of the adaptive network 15m based on the losses. That is, the processing unit 11 updates the parameters of the adaptive network 15m (such as the weights of each branch) using the adaptive network learning program 15c so that the combined loss of the distance loss between the feature amounts of the good-quality images and the distance loss between the feature amounts of the good-quality images and the pseudo-faulty images is reduced. Note that the combined loss of the two distance losses may be the sum of these distance losses, or may be the sum of values ​​obtained by multiplying each distance loss by a predetermined coefficient.

[0098] In this way, the adaptive network training unit 11c (see FIG. 3) trains the adaptive network 15m so as to increase the distance between the feature amounts of the good-quality images after feature conversion by the adaptive network 15m and the feature amounts of the pseudo-fault images after feature conversion by the adaptive network 15m. Feature conversion is performed so as to increase the distance between the feature amounts of the good-quality images and the feature amounts of the pseudo-fault images, thereby improving the discrimination accuracy during defect detection. In other words, the adaptive network training unit 11c (see FIG. 3) trains the adaptive network 15m so as to increase the degree of difference between the feature amounts of the good-quality images after feature conversion by the adaptive network 15m and the feature amounts of the pseudo-fault images after feature conversion by the adaptive network 15m. This is because the greater this degree of difference, the more appropriately good-quality images and pseudo-fault images can be discriminated.

[0099] <Effects> According to the second embodiment, the distance between the feature amounts of the good product image and the pseudo-fault image is used for training the adaptive network 15m, which prevents the feature amounts of the pseudo-fault image from being projected close to the feature amounts of the good product image in the feature amount conversion using the adaptive network 15m, thereby improving the accuracy of defect detection.

[0100] Third Embodiment The third embodiment differs from the first embodiment in the way adaptive network 15m (see FIG. 8) learns, but is otherwise similar to the first embodiment (such as the configuration of fault detection learning device 10: see FIG. 1). Therefore, only the parts that differ from the first embodiment will be described, and explanations of overlapping parts will be omitted.

[0101] FIG. 13 is a flowchart relating to learning of an adaptive network of the fault detection learning device according to the third embodiment (also see FIG. 1 as appropriate). The series of processes shown in Fig. 13 corresponds to step S105 (learning of the adaptive network 15m) in Fig. 6. In addition to step S105a in Fig. 13, the processes of steps S105b to S105f are the same as steps S105a to S105f in the first embodiment (see Fig. 11), and therefore description thereof will be omitted.

[0102] 13, after generating the features of the non-defective product image, the processing of the processing unit 11 proceeds to step S105k. In step S105k, the processing unit 11 calculates the reliability of the normal reference features. That is, the processing unit 11 calculates the reliability of the normal reference features based on the attribute information of the normal reference features from the non-defective product database 15g using the adaptive network learning program 15c.

[0103] Here, the "attribute information" of a normal reference feature refers to data related to the normal reference feature, such as the number of dimensions of the feature, the distance from the nearest other normal reference feature, and the distance from the cluster center of the normal reference feature. Furthermore, the "cluster center" refers to the coordinate value of the center (center of gravity) of a cluster when multiple normal reference features are clustered in feature space. For example, a predetermined calculation formula (reliability calculation formula) is pre-set so that the longer the distance between a normal reference feature and the cluster center, the lower the reliability. Alternatively, for example, the number of times a feature of a non-defective image is projected to the nearest normal reference feature during training of the adaptive network 15m may be used as attribute information. In this case, the higher the number of times, the higher the reliability of the normal reference feature.

[0104] In step S105m, the processing unit 11 determines, by the adaptive network learning program 15c, whether the reliability of the normal reference feature is equal to or less than a predetermined threshold. Here, the predetermined threshold is a reliability threshold that serves as a criterion for determining whether or not to exclude a certain normal reference feature, and is set in advance. For example, the predetermined threshold may be set to 0.

[0105] If the reliability of the normal reference feature is higher than the predetermined threshold in step S105m (step S105m: No), the processing unit 11 proceeds to step S105b. If the reliability of the normal reference feature is equal to or lower than the predetermined threshold (step S105m: Yes), the processing unit 11 proceeds to step S105n. In step S105n, the processing unit 11 uses the adaptive network learning program 15c to exclude low-reliability normal reference features whose reliability is equal to or less than a predetermined threshold. That is, the adaptive network learning unit 11c (see FIG. 3) excludes, from among multiple normal reference features, those whose reliability based on attribute information of the normal reference features is equal to or less than a predetermined threshold. The "attribute information" includes at least one of the number of dimensions of the normal reference feature, the distance between the normal reference feature and the nearest other normal reference feature, and the distance between the normal reference feature and the cluster center. By excluding low-reliability normal reference features in this way, the identification accuracy when detecting defects using the adaptive network 15m is improved.

[0106] In step S105p, the processing unit 11 adds predetermined normal reference features using the adaptive network learning program 15c. Here, the added normal reference features may be newly generated, or predetermined normal reference features previously stored in the non-defective product database 15g may be used as appropriate. In this manner, the adaptive network learning unit 11c (see FIG. 3) excludes and adds normal reference features based on the reliability of the attribute information of the normal reference features. For example, when the adaptive network learning unit 11c excludes from multiple normal reference features those whose reliability is equal to or less than a predetermined threshold, it adds the predetermined feature stored as a candidate for the normal reference feature as a new normal reference feature. This prevents the number of normal reference features from becoming too small. After performing the process of step S105p, the process proceeds to step S105b in the processing unit 11. As described above, the processes of steps S105b to S105f are the same as those in the first embodiment (see FIG. 11), and therefore, description thereof will be omitted.

[0107] <Effects> According to the third embodiment, normal reference features having a reliability equal to or less than a predetermined threshold are excluded, thereby improving the classification accuracy when detecting defects using the adaptive network 15 m. Furthermore, when a normal reference feature is excluded, a new normal reference feature is added, which prevents the number of normal reference features from becoming too small, and ultimately prevents a decrease in classification accuracy when detecting defects.

[0108] Fourth Embodiment The fourth embodiment differs from the first embodiment in that the number of defect variations of the target defect is set in multiple ways, and the number of defect variations with the highest detection performance is used for defect detection. Note that other aspects (such as the configuration of the defect detection learning device 10: see FIG. 1) are the same as those of the first embodiment. Therefore, only the parts that differ from the first embodiment will be described, and the description of the overlapping parts will be omitted.

[0109] FIG. 14 is a flowchart relating to a search for a fault condition in the fault detection learning device according to the fourth embodiment (also see FIG. 1 as appropriate). In step S201, the processing unit 11 sets the inspection object and a plurality of failure conditions. That is, the processing unit 11 receives information on the inspection object and information on the plurality of failure conditions input by operation via the input unit 13, and stores this information in the other information database 15h. As the plurality of failure conditions, for example, the number of failure variations of the target failure is set in a plurality of ways. For example, the first number of failure variations is set to 3, the second number of failure variations is set to 6, and the third number of failure variations is set to 9. Incidentally, the example in FIG. 5 shows a case where the number of failure variations is 3 (corresponding to the number of image patterns of pseudo failures D8, D9, and D10).

[0110] In step S202, the processing unit 11 repeats the processes of steps S203 to S205 until the value i starts from 1 and reaches the value I, which is the set number of bad conditions (three in the above example). In step S203, the processes of steps S102 to S108 (see FIG. 6) described in the first embodiment are executed in sequence. Next, in step S204, the processing unit 11 determines whether the detection performance of the pseudo-fault images in the i-th loop is equal to or greater than the provisional best performance. If the defect detection performance of the pseudo-fault images in the i-th loop is equal to or greater than the provisional best performance (step S204: Yes), the processing of the processing unit 11 proceeds to step S205.

[0111] In step S205, the processing unit 11 saves data corresponding to the provisional best performance. That is, the processing unit 11 saves the provisional classification network 15k, the provisional adaptive network 15m, and the provisional normal reference feature quantity by the defect detection program 15d. Then, the processing of the processing unit 11 proceeds to step S206. Also, in step S204, if the detection performance of the pseudo defect image in the ith loop is not equal to or greater than the provisional best performance (step S204: No), the processing of the processing unit 11 proceeds to step S206.

[0112] In step S206, if the value i has reached the value I, which is the set number of bad conditions, the processing unit 11 ends the loop of step S202. If the value i has not reached the value I, the processing unit 11 repeats the processes (steps S203 to S205) related to the loop of step S202.

[0113] In step S207, the processing unit 11 reads the data at the time of the highest performance. That is, the processing unit 11 reads the data of the classification network 15k, the adaptive network 15m, and the normal reference feature quantity corresponding to the highest performance by the defect detection program 15d. In this way, the classification network 15k and the adaptive network 15m are trained based on the one with the highest target defect detection performance among multiple candidates for the number of defect variations, which is the number of types of image patterns. Each data after training is used appropriately for detecting actual defective products.

[0114] <Effects> According to the fourth embodiment, the number of defect variations with the highest detection performance is identified from among multiple numbers of defect variations. That is, a search for defect conditions is performed for the number of defect variations that is difficult to set using a rule-based method (set by a human). This allows an appropriate number of defect variations to be set, thereby improving the identification accuracy when detecting defects.

[0115] Fifth Embodiment The fifth embodiment differs from the first embodiment in that the processing unit 11 determines whether to terminate learning of the adaptive network 15m (see FIG. 8) based on the integrated detection performance of a plurality of pseudo-fault images. Note that other aspects (such as the configuration of the fault detection learning device 10: see FIG. 1) are the same as those of the first embodiment. Therefore, only the parts that differ from the first embodiment will be described, and explanations of overlapping parts will be omitted.

[0116] FIG. 15 is a flowchart relating to calculation of the integrated detection performance in the fault detection learning device according to the fifth embodiment (also see FIG. 1 as appropriate). In step S301, the processing unit 11 sets an evaluation target of the pseudo-fault image. That is, the processing unit 11 stores input information of the evaluation target of the pseudo-fault image in the other information database 15h based on a user's operation via the input unit 13.

[0117] Next, in step S302, the processing unit 11 repeats the process of step S303 until the value j starts from 1 and reaches the value J, which is the number of pseudo-failure images to be evaluated including the target failure image. In step S303, the processing unit 11 evaluates the detection performance of the pseudo-fault image by the defect detection program 15d. The processing in step S303 is the same as the processing in step S106 in Fig. 6 except that pseudo-fault images other than the target defect are added to the evaluation targets for detection performance in the processing in step S106. That is, the processing unit 11 evaluates the detection performance of the pseudo-fault image based on the distance from the normal reference feature amount to the feature amount of the pseudo-fault image.

[0118] In step S304, if the value j reaches the value J of the number of pseudo-fault images to be evaluated, the processing unit 11 ends the loop of step S302. If the value j has not reached the value J, the processing unit 11 repeats the loop of step S302.

[0119] In step S305, the processing unit 11 calculates the integrated detection performance of the multiple pseudo-fault images using the defect detection program 15d. For example, if the detection performance of the multiple pseudo-fault images is well-balanced, the processing unit 11 increases the integrated detection performance, and conversely, if the detection performance is unbalanced, the processing unit 11 decreases the integrated detection performance. More specifically, a calculation formula for the integrated detection performance is preset so that the greater the extent to which the ratio of the detection performance value for the target defect to the detection performance value for pseudo-fault images other than the target defect deviates from a predetermined range, the lower the value. After performing the processing of step S305, the processing unit 11 ends the series of processes (END). In this way, the adaptive network learning unit 11c ends the learning of the adaptive network 15m based on the result of integrating the evaluation of the detection performance for the multiple pseudo-fault images. This allows the adaptive network learning unit 11c to appropriately determine whether to end the learning of the adaptive network 15m based on the integrated detection performance.

[0120] <Effects> According to the fifth embodiment, by evaluating the detection performance of pseudo-failure images other than target defects, it is possible to confirm that the detection performance of target defects is improved while the detection performance of other than target defects is maintained.

[0121] Sixth Embodiment The sixth embodiment differs from the first embodiment in that a plurality of target defects are set for the inspection object, and an identification network 15k (see FIG. 7) and an adaptive network 15m (see FIG. 8) are provided for each target defect. Note that other aspects (such as the configuration of the defect detection learning device 10: see FIG. 1) are the same as those of the first embodiment. Therefore, only the parts that differ from the first embodiment will be described, and a description of the overlapping parts will be omitted.

[0122] FIG. 16 is a flowchart showing the flow of processing related to defect detection in the defect detection learning device according to the sixth embodiment (see also FIG. 1 as appropriate). In step S401, the processing unit 11 sets a plurality of target defects for a predetermined inspection object. That is, the processing unit 11 stores data on the inspection object and the defect conditions including the plurality of target defects in the other information database 15h based on a user's operation via the input unit 13.

[0123] To give a specific example, a small-sized defect in portion 1a (see FIG. 5) of inspection object 1 is set as a first target defect, a small-sized defect in portion 1b (see FIG. 5) is set as a second target defect, and a small-sized defect in portion 1c (see FIG. 5) is set as a third target defect. In this way, at least one target defect is set for each of the multiple portions 1a, 1b, and 1c (see FIG. 5) of inspection object 1.

[0124] In step S402, the processing unit 11 repeats the process of step S403 until the value t starts from 1 and reaches the value T which is the target number of defects. In step S403, the processing unit 11 executes the same processes as steps S102 to S108 (see FIG. 6) in the first embodiment. Specifically, the discrimination network learning unit 11b (see FIG. 3) learns a discrimination network 15k (see FIG. 7) for each of a plurality of target failures. In addition, the adaptive network learning unit 11c (see FIG. 3) learns an adaptive network 15m for each of a plurality of target failures.

[0125] In step S404, if the value t reaches the value T that is the target number of defects, the processing unit 11 ends the loop of step S402. On the other hand, if the value t has not reached the value T, the processing unit 11 repeats the loop of step S402. In step S405, the processing unit 11 loops the process of step S406 until the value t starts from 1 and reaches the value T, which is the target number of defectives.

[0126] In step S406, the processing unit 11 detects defects in the test image by the defect detection program 15d. That is, the processing unit 11 performs defect detection on a predetermined test image obtained by image capture by the imaging device 20 based on the abnormality evaluation engine 15n, the classification network 15k, the adaptive network 15m, and the normal reference feature amount. Note that when detecting defects using the pseudo-defective image, the classification network 15k (see FIG. 7) and the adaptive network 15m (see FIG. 8) corresponding to each of a plurality of target defects are used.

[0127] In step S407, if the value t reaches the value T that is the target number of defects, the processing unit 11 ends the loop of step S405. On the other hand, if the value t has not reached the value T, the processing unit 11 repeats the loop of step S405. In step S408, processing unit 11 integrates and outputs the defect detection results of the test image using defect detection program 15d. For example, processing unit 11 causes display unit 14 to display the integrated detection performance based on each network that improves the detection performance of the first, second, and third target defects for each of portions 1a, 1b, and 1c (see FIG. 5) in this order.

[0128] In this way, the performance of defect detection is evaluated based on an integrated detection performance value obtained by integrating multiple detection performance values ​​for multiple target defects in a predetermined manner. The integrated detection performance value may be the sum of multiple detection performance values ​​for multiple target defects, or may be calculated based on another predetermined formula. For example, the sum of values ​​obtained by multiplying each of multiple detection performance values ​​for target defects by a predetermined coefficient may be used as the integrated detection performance value. After performing the process of step S408, the processing unit 11 ends the series of processes (END).

[0129] <Effects> According to the sixth embodiment, a target defect is set for each region of the object to be inspected, and detection performance is integrated, thereby further improving the discrimination accuracy when detecting defects.

[0130] Seventh Embodiment In the seventh embodiment, an image inspection system 100 (see Figure 17) is described, which includes a defect detection learning device 10 (see Figure 17) having a configuration similar to that of the first embodiment (see Figure 1), and an image inspection device 40 (see Figure 17) that performs image inspection of the inspection object 1 using the learning results of the defect detection learning device 10.

[0131] FIG. 17 is a functional block diagram of an image inspection system 100 according to the seventh embodiment. The image inspection system 100 shown in Fig. 17 is a system for determining whether an object to be inspected is good or bad, and for determining the type of defective portion of the object to be inspected, based on an image of the object to be inspected transmitted from an imaging device 20. As shown in Fig. 17, the image inspection system 100 is configured to include a defect detection learning device 10, an imaging device 20, and an image inspection device 40. Note that the defect detection learning device 10 and the imaging device 20 have the same configuration as those in the first embodiment (see Fig. 1), and therefore description thereof will be omitted.

[0132] 17 is a device that determines whether an object to be inspected is good or bad and determines the type of defective portion of the object to be inspected, based on a captured image of the object to be inspected transmitted from the imaging device 20. That is, the image inspection device 40 acquires a trained classification network 15k (see FIG. 7) and a trained adaptive network 15m (see FIG. 8) from the defect detection learning device 10. Then, the image inspection device 40 performs defect detection on the captured image of the object to be inspected, based on the acquired classification network 15k and adaptive network 15m.

[0133] 17, the image inspection device 40 includes a processing unit 41, a communication unit 42, an input unit 43, a display unit 44, and a storage unit 45. The functions of the processing unit 41, the communication unit 42, the input unit 43, and the display unit 44 are similar to the functions of the processing unit 11 (see FIG. 1, the same applies below), the communication unit 12, the input unit 13, and the display unit 14 of the defect detection learning device 10 described in the first embodiment, and therefore will not be described again. The hardware configuration of the image inspection system 100 may be similar to that shown in FIG. 2, for example.

[0134] 17, the communication unit 42 is connected to the imaging device 20 and also to the defect detection learning device 10. The data of the learning result of the defect detection learning device 10 is received from the defect detection learning device 10 via the communication unit 42.

[0135] 17, the storage unit 45 of the image inspection device 40 stores a defect detection program 45a, a GUI execution program 45b, an other information database 45c, a trained discrimination network 45d, and a trained adaptive network 45e. The defect detection program 45a performs defect detection on a captured image of an object to be inspected. The GUI execution program 45b accepts input of predetermined setting information based on a user's input operation. The other information database 45c stores data including normal reference features.

[0136] 17, the trained classification network 15k (see FIG. 7) in the defect detection learning device 10 is shown as classification network 45d of the image inspection device 40. Also, the trained adaptive network 15m (see FIG. 8) in the defect detection learning device 10 is shown as adaptive network 45e of the image inspection device 40.

[0137] The processing unit 41 of the image inspection device 40 detects defects in the object to be inspected and identifies the type of defect based on the classification network 45d, the adaptive network 15m, and the normal reference feature. That is, the processing unit 41 inputs a captured image to the classification network 45d and extracts feature values ​​from the captured image. The processing unit 41 then performs feature conversion using the adaptive network 45e and performs defect detection based on the distance between the converted feature value and the normal reference feature value. The defect detection results are displayed in a predetermined manner on the display unit 44.

[0138] <Effects> According to the sixth embodiment, the image inspection device 40 can perform defect detection of an inspection object with high accuracy based on the trained discrimination network 45d and adaptive network 45e.

[0139] <<Variations>> The defect detection learning device 10, image inspection device 40, and image inspection system 100 according to the present disclosure have been described in the above embodiments, but they are not limited to these descriptions and various modifications can be made. For example, in the first embodiment, the case where the parameters of the adaptive network 15m are updated based on the distance loss between feature amounts of non-defective images and the anomaly loss of non-defective images has been described (see FIG. 11), but the present invention is not limited to this. That is, the parameters of the adaptive network 15m may be updated based on at least one of the distance loss between feature amounts of non-defective images and the anomaly loss of non-defective images.

[0140] In addition, in each embodiment, the case where the defect variation of the target defect is given as a variation of the image pattern has been described, but this is not limited to this. That is, the defect variation may be generated based on at least one of the image pattern, position, and shape of the defect portion. For example, the processing unit 11 may extract defects with different shapes from a predetermined pattern image or texture image and combine them with a non-defective image.

[0141] As a method for changing the ratio of target defects to the total number of pseudo-failure images, for example, the number of variations of image patterns corresponding to the failure variations of target defects may be changed by operation via the input unit 13.

[0142] Furthermore, a plurality of types of engines may be used in combination as the abnormality degree evaluation engine 15n. Furthermore, the abnormality degree evaluation engine 15n is not limited to a rule-based engine, and a learning-type engine may also be used.

[0143] Furthermore, the respective embodiments can be combined as appropriate. For example, the first embodiment (see FIG. 11) and the second embodiment (see FIG. 12) may be combined, and the parameters of the adaptive network 15m may be updated based on the distance loss between the feature amounts of the non-defective images, the distance loss between the feature amounts of the non-defective images and the feature amounts of the pseudo-faulty images, and the abnormality map of the non-defective images. Alternatively, for example, the second or third embodiment may be combined with the fourth embodiment. The second to fourth embodiments may be combined with the fifth embodiment. The second to fifth embodiments may be combined with the sixth embodiment. The second to sixth embodiments may be combined with the seventh embodiment. Various other combinations are also possible.

[0144] A program for causing the defect detection learning device 10 (computer) to execute the defect detection learning method described in each embodiment can be provided via a communication line, or can be written to a recording medium 30 such as a CD-ROM (see Figure 2) and distributed.

[0145] The present disclosure is not limited to the above-described embodiments and includes various modifications. For example, the embodiments have been described in detail to clearly explain the present disclosure, and are not necessarily limited to those including all of the described configurations. Furthermore, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another example to the configuration of one embodiment. Furthermore, it is possible to add, delete, or replace part of the configuration of each embodiment with other configurations.

[0146] Furthermore, the above-mentioned configurations, functions, processing units, processing means, etc. may be partly or entirely implemented in hardware, for example, by designing them as integrated circuits. Furthermore, the above-mentioned configurations, functions, etc. may be implemented in software, with a processor interpreting and executing a program that implements each function. Information such as the programs, tables, and files that implement each function can be stored in a memory, a recording device such as a hard disk or SSD (Solid State Drive), or a recording medium such as an IC card, SD card, or DVD. In addition, the control lines and information lines shown are those that are considered necessary for the explanation, and do not necessarily show all the control lines and information lines in the product. In reality, it can be assumed that almost all components are interconnected. [Explanation of symbols]

[0147] 1. Inspection object 1a,1b,1c part 10 Defect detection learning device 11 Processing section 11a Pseudo defect image generation unit 11b Classification network training unit 11c Adaptive Network Learning Unit 11d Defect detection section 11e Display control unit 12 Communications Department 13 Input section 14 Display section 15 Storage section 15k Identification Network 15m adaptive network 20 Imaging device 30 Recording media G1 Good product image G2,G3,G4,G5,G6,G7,G8,G9,G10 Pseudo defective image Step S102 (pseudo-fault image generation step) Step S103 (classification network training step) S105 step (adaptive network learning step)

Claims

1. a pseudo-failure image generating unit that generates a pseudo-failure image of the inspection object using a non-defective product image of the inspection object; a classification network training unit configured to train a classification network to perform a classification task of distinguishing between the non-defective image and the pseudo-defective image; an adaptive network training unit that trains an adaptive network that performs predetermined feature transformation so as to adapt the normal reference feature of the non-defective product image extracted from the discrimination network to a defect detection task for the inspection object; a defect detection unit that detects defective products using a captured image of the inspection object based on the normal reference feature after adaptation to the defect detection task, The pseudo-failure image includes an image set as a target failure under a predetermined failure condition, A defect detection learning device, wherein the ratio between the number of pseudo-failure images of the target defect and the number of other pseudo-failure images different from the target defect is adjusted by operation via an input unit.

2. The number of the pseudo-failure images of the target failure is greater than the number of each type of the pseudo-failure images different from the target failure.

2. The fault detection learning device according to claim 1,

3. The pseudo-failure image is an image in which a predetermined image pattern is synthesized with a part of the non-defective image, the pseudo-failure image generating unit generates the pseudo-failure images of a plurality of types of the target defects for each of the image patterns; The shape of the contour of the image pattern is common in the pseudo-failure images of the plurality of types of target defects.

2. The fault detection learning device according to claim 1,

4. a display control unit that displays a pseudo-failure image of the target failure on a display unit and also displays a transition of the failure detection rate, which indicates the detection performance of the target failure, on the display unit; 2. The fault detection learning device according to claim 1,

5. The adaptive network learning unit learns the adaptive network so that the degree of abnormality of the non-defective image is equal to or less than a predetermined threshold.

2. The fault detection learning device according to claim 1,

6. The adaptive network learning unit performs defect detection on the pseudo-fault image of the target defect, and terminates learning of the adaptive network based on an evaluation of detection performance in the defect detection.

2. The fault detection learning device according to claim 1,

7. The adaptive network training unit trains the adaptive network so as to increase the degree of difference between the feature amounts of the non-defective image after feature transformation by the adaptive network and the feature amounts of the pseudo-defective image after feature transformation by the adaptive network.

2. The fault detection learning device according to claim 1,

8. The adaptive network learning unit excludes and adds the normal reference features based on the reliability of attribute information of the normal reference features.

2. The fault detection learning device according to claim 1,

9. The adaptive network learning unit terminates learning of the adaptive network based on a result of integrating evaluations of detection performance for a plurality of the pseudo-fault images.

2. The fault detection learning device according to claim 1,

10. The discrimination network and the adaptive network are trained based on the candidate with the highest detection performance for the target defect among a plurality of candidates for the number of defect variations, which is the number of types of the image patterns.

4. The fault detection learning device according to claim 3, wherein:

11. At least one target defect is set for each of a plurality of portions of the inspection object; the discrimination network learning unit learns the discrimination network for each of the plurality of target defects; the adaptive network training unit trains the adaptive network for each of the plurality of target defects; When detecting defects using the pseudo-fault image, the discrimination network and the adaptive network corresponding to each of the plurality of target defects are used, and further, performance evaluation of defect detection is performed based on an integrated detection performance value obtained by integrating a plurality of detection performance values ​​for the plurality of target defects.

2. The fault detection learning device according to claim 1,

12. An image inspection device that acquires the trained identification network and the trained adaptive network from the defect detection learning device described in claim 1, and detects defective products using captured images of the object to be inspected based on the identification network and the adaptive network.

13. The fault detection learning device according to claim 1, An image inspection system including an image inspection device that acquires the trained identification network and the trained adaptive network from the defect detection learning device, and detects defective products using captured images of the object to be inspected based on the identification network and the adaptive network.

14. a pseudo-failure image generating step in which a processing unit generates a pseudo-failure image of the inspection object using a non-defective image of the inspection object; a classification network training step in which the processing unit trains a classification network to perform a classification task of distinguishing between the non-defective image and the pseudo-defective image; an adaptive network training step in which the processing unit trains an adaptive network that performs predetermined feature transformation so as to adapt the normal reference feature of the non-defective product image extracted from the discrimination network to a defect detection task for the inspection object; a defect detection step of detecting a defective product using a captured image of the inspection object based on the normal reference feature after adaptation to the defect detection task, The pseudo-failure image includes an image set as a target failure under a predetermined failure condition, A defect detection learning method, in which a ratio between the number of pseudo-failure images of the target defect and the number of other pseudo-failure images different from the target defect is adjusted by operation via an input unit.

15. A program for causing a computer to execute the defect detection learning method according to claim 14.

Images