Defect inspection device and defect inspection method

Abstract

To improve the defect detection accuracy of an object in a focused process of inspection.SOLUTION: A defect inspection device that inspects a defect of an object in a focused process of inspection, acquires a training image that is an image of the object in the focused process of inspection, and further trains a base model neural network intended for feature extraction using the training images and data regarding defects visible in the training image, then acquires an inspection image that is an image of the object's target inspection area in the focused process of inspection, and uses an individual model obtained by additional training of the base model in a fine-tuning unit to evaluate a defect of the object in the focused process of inspection from the inspection image, evaluates a defect of the object in the focused process of inspection based on a feature quantity corresponding to the defect of the object depicted in the inspection image to perform defect inspection of the object based on the evaluation result from a first evaluation unit and the evaluation result from a second evaluation unit.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present disclosure relates to a defect inspection apparatus and a defect inspection method. [Background technology]

[0002] In many industrial fields, there is a strong movement to use classifiers trained by machine learning methods to extract experience and knowledge from large amounts of data and use them to automate processes. In particular, in the field of image recognition in non-destructive testing, such as surface inspection in the manufacturing industry and defect inspections such as ultrasonic flaw detection, many machine learning models based on neural networks, including deep learning, have been introduced, and dramatic improvements in accuracy have been confirmed. In order to train a machine learning model in a supervised manner, it is necessary to prepare a sufficient amount of data (hereinafter referred to as training data or teacher data) to which correct answers have been annotated by humans in advance.

[0003] However, in non-destructive testing such as surface inspection and ultrasonic flaw detection, the defects to be detected may occur infrequently or collecting data on defects may be operationally difficult, making it generally difficult to collect a sufficient amount of training data.

[0004] Therefore, a technique called fine tuning is sometimes used to obtain a certain level of accuracy while suppressing the amount of training data that is difficult to collect. Fine tuning is a technique in which, instead of randomly setting initial values ​​as is common in supervised learning, additional learning is performed using a machine learning model (pre-trained model) that has previously trained on a specific image data set as the initial value. For example, paragraph "0049" of Patent Document 1 states the following:

[0005] Next, I will explain fine-tuning. Fine-tuning is a technique that improves object recognition accuracy and requires less training data by reusing a trained convolutional neural network. A trained convolutional neural network is a hierarchical network trained on a large-scale dataset. There are various types of large-scale datasets, such as datasets for image classification, object detection, and segmentation.

[0006] Furthermore, paragraph "0017" of Patent Document 2 contains the following description regarding a trained model used for fine tuning.

[0007] The trained model may be a trained model of another semiconductor device or the like that has been optimized in an external learning device different from the learning device 13.

[0008] The technology described in Patent Document 2 uses for fine tuning a model that has been trained using some kind of data acquired from other semiconductor devices, rather than images that exist in nature or on the Internet. [Prior art documents] [Patent documents]

[0009] [Patent Document 1] Japanese Patent Publication No. 2022-66637 [Patent Document 2] Japanese Patent Application Publication No. 2023-106098 Summary of the Invention [Problem to be solved by the invention]

[0010] However, in surface inspection and ultrasonic flaw detection in manufacturing inspection processes, fine-tuned machine learning models are not necessarily effective for all defects to be detected. For example, there are defects for which conventional inspection flows that do not use machine learning models, such as classical image processing methods such as binarization and labeling, or inspection flows that use empirically designed image feature-based discrimination processing and conditional expressions, are more effective. In other words, a fine-tuned machine learning model alone, such as that described in Patent Document 1, cannot accurately detect all target defects.

[0011] Furthermore, the technology described in Patent Document 2 does not specify a specific method for collecting data for fine tuning. Furthermore, only models obtained by fine-tuning trained models are considered, and similar to the technology described in Patent Document 1, it is not possible to detect all of the defects to be detected.

[0012] In recent years, a concept called a foundational model has been proposed in the field of machine learning. A foundational model is a trained model that is trained using an extremely large amount of data from diverse domains, and even multimodal data, that is incomparable to the training data used in previous trained models. The data used to train the foundational model can be any type of data that can be treated as digital data, such as images, audio, text, and numerical values.

[0013] For example, the foundation model performs weighting calculations (using machine learning techniques) on input training data while imposing certain constraints, optimizing the weighting at each layer of the machine learning model. An example of optimization is restoring the output results to be identical to the original training data (minimizing the restoration error). Therefore, a trained foundation model is expected to have the ability to extract the features of input data from the data. Furthermore, the foundation model is considered to be able to handle a variety of tasks simply by additional learning with a very small amount of training data, i.e., fine-tuning. Note that the learning of the foundation model is not limited to supervised learning, and unsupervised learning may also be used.

[0014] However, when defect inspection is performed using only a model that is a fine-tuned version of the base model, or when defects in an object are inspected from feature quantities corresponding to the defects in the object captured in the inspection image, as in the conventional inspection flow, it may not be possible to detect the defect accurately depending on the type of defect.

[0015] The present disclosure has been made in consideration of the above circumstances, and provides a defect inspection device and a defect inspection method that improve the accuracy of defect detection, specifically, that enables the suppression of overdetection and non-detection of defects, compared to when defect inspection is performed using only a model that is a fine-tuned base model, or when defect inspection is performed from feature quantities corresponding to defects in an object captured in an inspection image. [Means for solving the problem]

[0016] In order to solve the above problem, according to one aspect of the present disclosure, there is provided a defect inspection device that inspects an object for defects in an inspection process of interest, the defect inspection device having: a teacher image acquisition unit that acquires a teacher image of the object in the inspection process of interest; a fine tuning unit that additionally trains a base model, which is a neural network for the purpose of feature extraction, using the teacher image and data related to the defects shown in the teacher image; an inspection image acquisition unit that acquires an inspection image of a portion of the object to be inspected in the inspection process of interest; a first evaluation unit that evaluates the defect of the object in the inspection process of interest from the inspection image using an individual model additionally trained by the fine tuning unit on the base model; a second evaluation unit that evaluates the defect of the object in the inspection process of interest from feature quantities corresponding to the defect of the object shown in the inspection image; and a defect inspection unit that inspects the defect of the object based on the evaluation results of the first evaluation unit and the evaluation results of the second evaluation unit.

[0017] In order to solve the above problem, according to one aspect of the present disclosure, there is provided a defect inspection method for inspecting an object for defects in an inspection process of interest, the defect inspection method comprising: a teacher image acquisition step for acquiring a teacher image of the object in the inspection process of interest; a fine-tuning step for additionally training a base model, which is a neural network for the purpose of feature extraction, using the teacher image and data related to the defects shown in the teacher image; an inspection image acquisition step for acquiring an inspection image of a portion of the object to be inspected in the inspection process of interest; a first evaluation step for evaluating the defect of the object in the inspection process of interest from the inspection image using an individual model additionally trained from the base model in the fine-tuning step; a second evaluation step for evaluating the defect of the object in the inspection process of interest from feature quantities corresponding to the defect of the object shown in the inspection image; and a defect inspection step for inspecting the object for defects based on the evaluation results of the first evaluation step and the evaluation results of the second evaluation step. [Effects of the Invention]

[0018] According to the present disclosure, it is possible to improve the accuracy of defect detection, specifically to reduce overdetection and non-detection of defects, compared to when defect detection is performed using either a model that is a fine-tuned version of a base model alone, or when defect inspection is performed from feature quantities corresponding to defects in an object captured in an inspection image. [Brief explanation of the drawings]

[0019] [Figure 1] FIG. 2 is a block diagram showing a hardware configuration of the defect inspection apparatus. [Figure 2] FIG. 2 is a block diagram showing the configuration of functions realized by a CPU of the defect inspection device. [Figure 3] 1 is a flowchart of a defect inspection method. [Figure 4] FIG. 2 is a diagram illustrating an example of the configuration of an individual model. DETAILED DESCRIPTION OF THE INVENTION

[0020] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings.

[0021] First Embodiment

[0022] FIG. 1 shows the hardware configuration of a defect inspection apparatus 10 according to the present disclosure.

[0023] The defect inspection device 10 is a device that inspects an object for defects in an inspection process of interest. The inspection process of interest is a process in which the object to be inspected is inspected for defects, and examples thereof include a surface inspection process in which the surface properties of the object are inspected, and an ultrasonic inspection process in which internal defects of the object are inspected, in a predetermined order.

[0024] The object is an object to be inspected when the defect inspection device 10 inspects the object for defects, and can be, for example, a strip-shaped steel plate such as a rolled steel plate or a slab in a steel mill, etc., but is not limited to this. The object may be made of other materials than iron, such as aluminum or titanium, and may have various shapes other than a flat strip-like shape, such as a columnar billet, steel bar, wire rod, tubular steel pipe, or H-section steel.

[0025] Furthermore, defects include, for example, unevenness or scratches on the surface of an object, cavities inside the object, etc., but are not limited to these, and may also be various non-stationary parts in a product, such as stains or roughness abnormalities on the surface, internal grain boundaries, etc.

[0026] 1, the defect inspection device 10 includes a controller 12. The controller 12 is configured as a device including a general computer.

[0027] The controller 12 includes a central processing unit (CPU) 12A, a read-only memory (ROM) 12B, a random access memory (RAM) 12C, and an input / output interface (I / O) 12D. The CPU 12A, ROM 12B, RAM 12C, and I / O 12D are connected to each other via a bus 12E. The bus 12E includes a control bus, an address bus, and a data bus.

[0028] Furthermore, an operation unit 14, a display unit 16, a communication unit 18, and a storage unit 20 are connected to the I / O 12D.

[0029] The operation unit 14 includes, for example, a mouse and a keyboard, and is a mechanism for inputting information to the defect inspection device 10 in accordance with the operator's intentions.

[0030] The display unit 16 is configured by, for example, a liquid crystal display, and is a mechanism for conveying the progress and final results of the processing performed by the defect inspection device 10 to an operator or the like by displaying them.

[0031] The communication unit 18 is an interface for performing data communication with an external device such as an external server.

[0032] The storage unit 20 is configured with a non-volatile external storage device such as a hard disk. As shown in Fig. 1, the storage unit 20 stores a defect inspection program 22, a base model 24, an individual model 24A, fine-tuning teacher data 26, and inspection images 28. The base model 24, fine-tuning teacher data 26, and inspection images 28 may be prepared in advance and stored in the storage unit 20, or may be obtained from an external server (not shown).

[0033] Fig. 2 is a block diagram showing the configuration of functions realized by CPU 12A of defect inspection apparatus 10. As shown in Fig. 2, CPU 12A functionally includes the following functional units: a teacher image acquisition unit 30, a fine tuning unit 32, an inspection image acquisition unit 34, a first evaluation unit 36, a second evaluation unit 38, and a defect inspection unit 40.

[0034] The CPU 12A reads and executes the defect inspection program 22 stored in the storage unit 20, thereby functioning as each functional unit shown in FIG.

[0035] The teacher image acquisition unit 30 acquires a teacher image 26A obtained by imaging an object in a target inspection process.

[0036] The teacher images 26A are, for example, images of the surface of an object captured in a surface inspection process or A-scope images, B-scope images, C-scope images, etc. acquired in an ultrasonic flaw detection inspection process, and can include not only images containing defects such as scratches, but also images that do not contain defects. Furthermore, the teacher images 26A are not limited to images acquired in an actual inspection process, and can also be images generated by simulation or images generated by a machine learning model capable of image generation.

[0037] The fine tuning unit 32 is a functional unit that performs additional learning on the basic model 24, which is a neural network for the purpose of feature extraction, using fine tuning teacher data 26 that includes teacher image 26A and data 26B regarding defects captured in teacher image 26A.

[0038] The foundation model 24 is a machine learning model trained using a large amount of training data, and is a neural network for feature extraction. As a machine learning model, the foundation model 24 may have a classification structure (evaluation is performed on a per-image basis when the data is an image) or a segmentation structure (evaluation is performed on a per-pixel basis when the data is an image).

[0039] The training data used during training does not necessarily have to be data related to the manufacturing process of an object, such as a steel product, or the inspection of defects in the object. Images commonly found in nature or images on the Internet can be used, and general-purpose images can be used. For example, publicly available images, explanatory text, vector values, audio data, and other data that can be handled as digital data can be used without limiting the type or form. Therefore, data obtained from an inspection process other than the target inspection process or from a process that performs processing other than inspection can also be used as training data. Using such data allows the first evaluation unit 36, described below, to learn using data that is relatively similar to the inspection image 28 (compared to general-purpose data), thereby reducing the processing load on the first evaluation unit 36.

[0040] The base model 24, for example, calculates input learning data while imposing some constraints, and optimizes each layer of the machine learning model. Optimization, for example, can be performed by unsupervised learning to minimize the recovery error. After learning, the base model 24 has the function of extracting the characteristic parts of the input data from the data.

[0041] The fine-tuning unit 32 is assumed to have acquired in advance the base model 24, which has learned from such a large amount of training data and is capable of extracting features from the data, by obtaining it from outside the defect inspection device 10 or by creating it in-house using a large amount of training data within the defect inspection device 10.

[0042] The fine-tuning unit 32 acquires data 26B about the defects shown in the teacher image 26A, which indicates the characteristics of the defects shown in the teacher image 26A, such as the type, degree of harmfulness, position, occurrence cycle, etc., of the defects shown in the teacher image 26A, by having an inspector or the like inspect the teacher image 26A (i.e., an image of the object in the inspection process of interest) acquired by the teacher image acquisition unit 30 or by making a judgment using a program. The data 26B about the defects shown in the teacher image 26A is also stored in the memory unit 20 as fine-tuning teacher data 26 together with the teacher image 26A.

[0043] The fine tuning unit 32 generates an individual model 24A corresponding to a specific task by additionally training the base model 24 using fine tuning training data 26 including a training image 26A and data 26B regarding defects captured in the training image 26A.

[0044] That is, the base model 24 previously acquired by the fine-tuning unit 32 can handle various types of training data by learning from a large amount of training data regardless of data format. Furthermore, it can extract features contained in the training data from the various training data. However, the object to be inspected by the defect inspection device 10 according to this embodiment is a defect of an object in an inspection process of interest. The base model 24 does not generally learn about individual objects that exhibit unique behavior for each process, i.e., defects of an object in an inspection process of interest. Therefore, even if an attempt is made to obtain some inference result about a defect of an object in an inspection process of interest using the base model 24 as is, it is considered that an accurate result will not be obtained. Therefore, in this embodiment, instead of using the general-purpose base model 24 previously acquired by the fine-tuning unit 32 as is, processing is performed to apply the base model 24 to defects in the inspection process of interest by using information about the inspection process of interest.

[0045] Specifically, additional learning is performed by inputting a teacher image 26A, which is an image of an object in an inspection process of interest, and data 26B about defects captured in the teacher image 26A (information that an inspector or the like believes to be correct about the defect), into a general-purpose base model 24, so that the feature extraction function of the base model 24 can be applied to defects and other phenomena specific to the inspection process of interest (fine tuning).The fine-tuned base model 24 will be referred to as an individual model 24A.

[0046] The inspection image acquisition unit 34 is a functional unit that acquires an inspection image 28, which is an image of a portion of an object to be inspected in a focused inspection process. The inspection image 28 is an image of an area including the portion to be inspected on the surface of the object in the inspection process, similar to the teacher image 26A, or an A-scope image, B-scope image, C-scope image, or the like acquired by inspecting the area including the portion in an ultrasonic flaw detection inspection process, for the object that is the object to be inspected in the focused inspection process.

[0047] The inspection image acquisition unit 34 can acquire the inspection image 28 by imaging a desired inspection area of ​​an object to be inspected using an imaging unit (not shown) provided in the inspection process of interest, and imaging the image as the inspection image 28. Note that the inspection image 28 may also be acquired by reading out the inspection image 28 stored in the storage unit 20 in advance.

[0048] The first evaluation unit 36 ​​is a functional unit that evaluates defects of an object in a target inspection process from the inspection image 28 using the individual model 24A obtained by additional learning of the base model 24 by the fine tuning unit 32. That is, the first evaluation unit 36 ​​inputs the inspection image 28 to the individual model 24A generated by the fine tuning unit 32 and acquires defect information output from the individual model 24A. For example, if the individual model 24A is a neural network with a structure for semantic segmentation, defect information for each pixel of the inspection image 28 is output. If the individual model 24A is a neural network with a structure for classification, defect information for each image is output.

[0049] The second evaluation unit 38 is a functional unit that evaluates defects in an object in an inspection process of interest from feature amounts corresponding to the defects in the object captured in the inspection image 28. That is, unlike the first evaluation unit 36, the second evaluation unit 38 does not perform evaluation using a machine learning model, but performs a conventional evaluation that has traditionally been used in inspection processes, such as extracting feature amounts (physical feature amounts such as size, position, shape, brightness, color tone, etc.) of the defects captured in the captured image from the captured image and evaluating the type and harmfulness of the defect from the obtained feature amounts.

[0050] Specifically, it is desirable that the second evaluation unit 38 performs known image processing such as preprocessing such as noise removal, binarization, and labeling on the inspection image 28. Based on the information after labeling, the defect information may be determined using a machine learning model such as SVM (Support Vector Machine), or the defect information may be determined on a rule-based basis using domain knowledge related to operations such as conditional expressions.

[0051] The defect inspection unit 40 is a functional unit that inspects the object for defects based on the evaluation results from the first evaluation unit 36 ​​and the evaluation results from the second evaluation unit 38. Furthermore, the defect inspection unit 40 may inspect the object for defects using only the defect information that is the evaluation result from the first evaluation unit 36, or may inspect the object for defects using only the defect information that is the evaluation result from the second evaluation unit 38.

[0052] Specifically, the defect inspection unit 40 performs a final evaluation of defects in the object by appropriately combining the results of the first evaluation unit 36, which performs evaluation using the individual model 24A, and the results of the second evaluation unit 38, which performs a conventional evaluation using feature amounts directly obtained from the captured image. Therefore, even if the evaluation methods used by the first evaluation unit 36 ​​and the second evaluation unit have advantages and disadvantages depending on the type of defect, etc., a highly accurate evaluation can be performed as the final evaluation.

[0053] In addition, when the individual model 24A has a structure for classification, if the evaluation by the first evaluation unit 36 ​​and the evaluation by the second evaluation unit 38 match, they may be used as the evaluation result of the defect inspection unit 40 as is. If they do not match, it is advisable to decide in advance which judgment should be prioritized. In cases such as when the individual model 24A has a structure for segmentation, the same processing as when the individual model 24A has a structure for classification can be performed on a pixel-by-pixel basis (for each pixel).

[0054] Furthermore, the defect inspection unit 40 may weight the evaluation results from the first evaluation unit 36 ​​and the evaluation results from the second evaluation unit 38 by a predetermined value that can take 0, and inspect the object for defects based on the results obtained by performing an AND operation or an OR operation. More specifically, the defect inspection unit 40 may weight the evaluation results from the first evaluation unit 36 ​​and the evaluation results from the second evaluation unit 38 by a predetermined value that can take 0 (i.e., including a case where only one of the first evaluation unit 36 ​​and the second evaluation unit 38 is used), and when both the first evaluation unit 36 ​​and the second evaluation unit 38 are used, inspect the object for defects based on the results obtained by performing an AND operation or an OR operation.

[0055] Specifically, taking as an example a case where the individual model 24A has a structure for classification, the evaluation result by the first evaluation unit 36, i.e., the defect information for each pixel of the inspection image 28 obtained by the individual model 24A, is defined as E1, and the evaluation result by the second evaluation unit 36, i.e., the defect information for each pixel of the inspection image 28 using the feature amount directly obtained from the inspection image 28, is defined as E2, with W1 being the weight assigned to the evaluation result by the first evaluation unit 36 ​​and W2 being the weight assigned to the evaluation result by the second evaluation unit 38. When inspecting the object for defects by performing an AND (logical product) operation on the evaluation result by the first evaluation unit 36 ​​and the evaluation result by the second evaluation unit 36, the defect information E for each pixel is calculated using the following equation (1).

[0056] E=(E1×W1)×(E2×W2) ···(1)

[0057] Here, the weights W1 and W2 can take 0 or a positive value. Therefore, when W1≠0 and W2=0, the defect information E is information that does not take into account the evaluation result from the second evaluation unit 38, but only takes into account the evaluation result from the first evaluation unit 36. Also, when W1=0 and W2≠0, the defect information E is information that does not take into account the evaluation result from the first evaluation unit 36, but only takes into account the evaluation result from the second evaluation unit 38. Also, when W1≠0 and W2≠0, the defect information E is information in which the evaluation result from the first evaluation unit 36 ​​and the evaluation result from the second evaluation unit 38 are weighted by the values ​​of the weights W1 and W2.

[0058] Furthermore, when inspecting an object for defects by performing an OR (logical sum) operation on the evaluation results from the first evaluation unit 36 ​​and the evaluation results from the second evaluation unit 36, defect information E for each pixel is calculated using the following equation (2).

[0059] E = (E1 × W1) + (E2 × W2) (2)

[0060] The weights W1 and W2 in the above formula (2) are the same as the weights W1 and W2 in the above formula (1).Which of the above formulas (1) and (2) should be used can be selected appropriately depending on the type of defect detected, etc.

[0061] Next, a description will be given of the processing of the defect inspection method executed by the CPU 12A of the defect inspection apparatus 10. A flowchart of the defect detection method according to the present disclosure is shown in Fig. 3. The CPU 12 loads a defect inspection program stored in the storage unit 20 into the RAM 12C and executes it, thereby executing the processing of the defect inspection method shown in Fig. 3.

[0062] In step S100, the CPU 12A reads the fine-tuning teacher data 26 from the storage unit 20 to obtain the teacher image 26A.

[0063] In step S101, the CPU 12A generates an individual model 24A by additionally learning, i.e., fine-tuning, the base model 24 stored in the memory unit 20 using the teacher image 26A acquired in step S100 and data 26B regarding defects obtained by having an inspector inspect the teacher image 26A.

[0064] The processing of steps S100 and S101 does not need to be performed every time and only needs to be performed once. However, if you want to perform fine tuning again using new fine tuning teacher data 26, you can perform the processing of steps S100 and S101 using the new fine tuning teacher data 26.

[0065] In step S102, CPU 12A acquires inspection image 28, which is an image of a region to be inspected of an object that is an object to be inspected in the inspection process of interest.

[0066] In step S103, the CPU 12A performs a first evaluation process. Specifically, using the individual model 24A generated in step S102, the CPU 12A evaluates defects of the object in the target inspection process from the inspection image 28. That is, the CPU 12A inputs the inspection image 28 to the individual model 24A, and acquires the defect information for each pixel output from the individual model 24A as the first evaluation result.

[0067] In step S104, CPU 12A performs a second evaluation process. Specifically, the CPU 12A evaluates defects in the target object in the inspection process of interest based on feature amounts corresponding to the defects in the target object captured in inspection image 28. For example, predetermined image processing is performed on inspection image 28 acquired in step S103, and a binarization process is performed on the image after the image processing to generate a binarized image, and feature amount extraction processing is performed on the binarized image. Next, the extracted feature amounts are input to a machine learning model such as SVM, and defect information for each pixel is acquired as the second evaluation result.

[0068] In step S105, the CPU 12A inputs the defect information E1 for each pixel, which is the first evaluation result obtained in step S103, and the defect information E2 for each pixel, which is the second evaluation result obtained in step S104, into the above formula (1) or (2), thereby obtaining the defect information E for each pixel. This provides the defect information E for each pixel of the inspection image 28.

[0069] In step S106, the CPU 12A obtains defect information including at least one of the defect position, defect type, defect position, defect harmfulness, crystal orientation, defect shape, and defect density based on the defect information E of each pixel obtained in step S105, and outputs the information to the display unit 16 for display.

[0070] In this manner, in the present disclosure, an individual model 24A obtained by fine-tuning a base model 24 using fine-tuning teacher data 26 is used to evaluate defects in an object from an inspection image 28, and the object's defects are also evaluated from feature quantities corresponding to the defects of the object captured in the inspection image 28, and the object's defects are inspected based on the results of both evaluations.

[0071] Therefore, by reusing the already established base model 24 and using a relatively small number of training images, it is possible to easily generate an individual model 24A that corresponds to the inspection process of interest and has feature extraction functions, thereby eliminating the need to go to the trouble of collecting a large amount of training data just for the inspection process of interest, and using the generated individual model 24, it is possible to achieve highly accurate defect evaluation that is suitable for the inspection process of interest.

[0072] Furthermore, by combining and weighting the evaluations of the first evaluation section 36 and the second evaluation section 38, the accuracy of defect detection can be improved even if the evaluations of the first evaluation section 36 and the second evaluation section 38 have strengths and weaknesses depending on the type of defect, etc.

[0073] Furthermore, since the individual model 24A is generated by fine-tuning the base model 24, model deployment, in which the individual model 24A is applied to an actual inspection process, can be performed quickly and frequently.

[0074] Furthermore, the base model 24 can also be configured as a neural network that has been trained with training data that includes at least one of data obtained from an inspection process other than the inspection process of interest and data obtained from another process that performs processing other than inspection. For example, since the base model 24 is trained including data obtained from other bases or processes that are not included in the own base or process, it is possible to improve the accuracy of defect detection even with fine tuning using a small amount of training data.

[0075] Furthermore, the evaluation can be shared between the evaluation results of the first evaluation unit 36, which evaluates defects in the object from the inspection image 28 using the individual model 24A obtained by fine-tuning the base model 24, and the evaluation results of the second evaluation unit 38, which evaluates defects in the object from feature quantities corresponding to the defects of the object captured in the inspection image 28. This allows the number of types of defects targeted by the individual model 24A to be reduced, simplifying the entire inspection process. Furthermore, it is possible to avoid the inspection becoming a black box, which would be caused by leaving defect inspection entirely to the individual model.

[0076] Second Embodiment

[0077] Next, a second embodiment will be described. In the second embodiment, the same parts as those in the first embodiment are denoted by the same reference numerals, and detailed description thereof will be omitted.

[0078] In the second embodiment, the processing of the first evaluation unit 36 ​​and the defect inspection unit 40 differs from that in the first embodiment.

[0079] In the first embodiment, the first evaluation unit 36 ​​acquired the output of the individual model 24A as defect information, but in the second embodiment, the first evaluation unit 36 ​​acquires data from the intermediate layer of the individual model 24A. Then, the defect inspection unit 40 inspects the object for defects based on the data obtained from the intermediate layer of the base model 24 in the evaluation by the first evaluation unit 36 ​​and the evaluation result by the second evaluation unit 38.

[0080] 4 is a diagram showing an example of the configuration of the individual model 24A. As shown in FIG. 4, the individual model 24A includes an input layer IL, n intermediate layers ML1 to MLn, and an output layer OL. The pixel values ​​of each pixel in the inspection image 28 are input to the input layer IL. The intermediate layers ML1 to MLn perform a predetermined weighting operation on the data input from the previous layer and output the data to the subsequent layer. The output layer OL outputs defect information related to defects in the object shown in the inspection image 28.

[0081] The data output from the intermediate layers ML1 to MLn is a feature amount that represents the feature of the test image .

[0082] The first evaluation unit 36 ​​acquires a feature value output from at least one of the hidden layers ML1 to MLn. In the example of Fig. 4, the feature value is acquired from the last hidden layer MLn, but the first evaluation unit 36 ​​may acquire a feature value from another hidden layer or may acquire a feature value from multiple hidden layers.

[0083] The defect inspection unit 40 inputs the feature amounts acquired by the first evaluation unit 36 ​​from at least one of the intermediate layers ML1 to MLn to a machine learning model such as SVM used in the second evaluation unit 38. As a result, the feature amounts acquired from at least one of the intermediate layers ML1 to MLn, in addition to the feature amounts of each pixel of the inspection image 28, are input to the machine learning model such as SVM used in the second evaluation unit 38. As a result, the machine learning model such as SVM outputs defect information that takes into account both the feature amounts of the inspection image 28 extracted by the individual model 24A obtained by fine-tuning the base model 24 and the feature amounts corresponding to defects in the object depicted in the inspection image. This improves the accuracy of defect detection compared to when defect inspection is performed using only a model obtained by fine-tuning the base model or when defect inspection is performed using feature amounts corresponding to defects in the object depicted in the inspection image.

[0084] The present disclosure is not limited to the above-described embodiments, and various modifications and applications are possible within the scope of the gist of the present disclosure.

[0085] Furthermore, the configuration of the defect inspection device 10 described in the above embodiment (see Figure 1) is just one example, and it goes without saying that unnecessary parts may be deleted or new parts may be added within the scope of the present disclosure. [Explanation of symbols]

[0086] 10 Defect inspection equipment 12 Controllers 14 Control section 16 Display 18 Communications Department 20 Memory section 22 Defect Inspection Program 24 Foundation Model 24A Individual Model 26 Training data for fine tuning 26A Teacher image 26B Defect Data 28 Inspection images 30 Image acquisition unit for teachers 32 Fine Tuning Section 34 Inspection image acquisition unit 36 First Evaluation Section 38 Second Evaluation Section 40 Defect Inspection Department

Claims

1. A defect inspection apparatus that inspects an object for defects in an inspection process of interest, a teacher image acquisition unit that acquires a teacher image obtained by imaging the object in the target inspection process; a fine-tuning unit that performs additional learning on a base model, which is a neural network for the purpose of feature extraction, using the training image and data related to the defect shown in the training image; an inspection image acquisition unit that acquires an inspection image that is an image of a portion of the object to be inspected in the inspection step of interest; a first evaluation unit that evaluates defects of the object in the target inspection process from the inspection image using an individual model obtained by additionally learning the base model in the fine tuning unit; a second evaluation unit that evaluates a defect of the object in the target inspection process based on a feature amount corresponding to the defect of the object captured in the inspection image; a defect inspection unit that inspects the object for defects based on the evaluation results of the first evaluation unit and the evaluation results of the second evaluation unit; A defect inspection device comprising:

2. The base model is 2. The defect inspection device according to claim 1, which is a neural network that has learned training data that includes at least one of data obtained from an inspection process other than the inspection process of interest and data obtained from another process that performs processing other than inspection.

3. The defect inspection unit 3. The defect inspection device according to claim 2, wherein the evaluation results from the first evaluation unit and the evaluation results from the second evaluation unit are each weighted by a predetermined value that can be 0, and the object is inspected for defects based on the results obtained by performing an AND operation or an OR operation.

4. The defect inspection unit 3. The defect inspection device according to claim 2, wherein the object is inspected for defects based on data obtained from an intermediate layer of the individual model in the evaluation by the first evaluation unit and the evaluation results by the second evaluation unit.

5. A defect inspection method for inspecting an object for defects in an inspection process of interest, comprising: a teacher image acquisition step of acquiring a teacher image obtained by imaging the object in the target inspection process; a fine-tuning step of additionally learning a base model, which is a neural network for feature extraction, using the training image and data related to the defect shown in the training image; an inspection image acquisition step of acquiring an inspection image that is an image of a portion of the object to be inspected in the inspection process of interest; a first evaluation step of evaluating defects of the object in the target inspection process from the inspection image using an individual model obtained by additionally learning the base model in the fine tuning step; a second evaluation step of evaluating defects of the object in the target inspection process based on feature amounts corresponding to defects of the object captured in the inspection image; a defect inspection step of inspecting the object for defects based on the evaluation results in the first evaluation step and the evaluation results in the second evaluation step; A defect inspection method comprising:

6. The base model is 6. The defect inspection method according to claim 5, wherein the neural network is trained with training data including at least one of data obtained from an inspection process other than the inspection process of interest and data obtained from another process performing processing other than inspection.

7. The defect inspection step includes:

7. The defect inspection method according to claim 6, wherein the evaluation results in the first evaluation step and the evaluation results in the second evaluation step are each weighted by a predetermined value that can be 0, and the object is inspected for defects based on the results obtained by performing an AND operation or an OR operation.

8. The defect inspection step includes:

7. The defect inspection method according to claim 6, wherein the object is inspected for defects based on data obtained from an intermediate layer of the individual model in the evaluation in the first evaluation step and the evaluation results in the second evaluation step.

Images