Method and System for Training an Automatic Defect Classification Inspection Device

Abstract

A computer-implemented method, an inspection system, and a non-transitory storage medium are provided for automatically generating a defect classification model using machine learning for inspecting semiconductor components and / or printed circuit board (PCB) components. The defect classification model is automatically constructed from a first combination of a trained neural network binary classifier and an optimizer and a second combination of a trained neural network multi-class classifier and an optimizer.

Description

Technical Field

[0001] This technical field generally relates to inspection apparatuses and methods for performing automatic defect inspection, and more particularly, to methods and systems for automatically classifying defects in products to be inspected. The methods and systems presented below are particularly suitable for the inspection of semiconductor products.

Background Art

[0002] The manufacturing process generally includes automatically inspecting the manufactured parts at individual milestones during the process, typically at least at the end of the manufacturing process. The inspection may be performed using an inspection apparatus that optically analyzes the manufactured parts to detect defective parts. Various techniques may be used, such as a camera combined with laser triangulation and / or laser interferometry. The automatic inspection apparatus ensures that the manufactured parts meet the expected quality standards and provides useful information regarding the adjustments that may be required for the manufacturing tools, manufacturing equipment, and / or manufactured parts, depending on the type of detected defect.

[0003] In the semiconductor industry, it is common for different types of parts to share the same manufacturing line, whether the target customers are the same or different. Therefore, the inspection apparatus needs to be able to distinguish between defective and non-defective parts and identify the types of defects contained in the detected defective parts. Since classifying defects is often labor-intensive, the involvement of experts in the inspection apparatus and manufacturing process is required so that the apparatus can be adjusted and configured to appropriately detect defects. In most cases, it is necessary to take the apparatus offline to configure an inspection apparatus for adjusting existing types of defects or detecting new types of defects. Well-known defect detection methods in the semiconductor industry include comparing the captured image with a "mask" or "optimal part layout", but this method misses many undetected defects.

Summary of the Invention

Problems to be Solved by the Invention

[0004] There is a need for inspection apparatuses and methods that can help improve or facilitate the process of classifying defects when automatically inspecting products.

Means for Solving the Problem

[0005] According to one aspect, there is provided a computer-implemented method for automatically generating a defect classification model using machine learning for use in an automatic inspection apparatus for inspecting manufactured parts. The method includes the step of obtaining an inspection image of a part imaged by an inspection apparatus. The inspection image is associated with label information indicating whether a given image corresponds to a non-defective product or a defective product for inspection images corresponding to defective products such as semiconductor parts and / or printed circuit board (PCB) parts, and further indicating the defect type if the inspection image corresponds to a defective product.

[0006] The method further includes the step of training a binary classifier using a first subset of the inspection images to determine whether the inspection image corresponds to a non-defective product or a defective product. The binary classifier uses a first combination of a neural network architecture and an optimizer. The binary classifier is trained by repeatedly updating the weights of the nodes in different layers of the neural network architecture used in the first combination.

[0007] The method further includes the step of training a multi-class classifier using a second subset of the inspection images corresponding to defective products to identify the defect type in the inspection images previously determined to correspond to defective products by the binary classifier. The multi-class classifier uses a second combination of a neural network architecture and an optimizer. The multi-class classifier is trained by repeatedly updating the weights of the nodes in different layers of the neural architecture in the second combination.

[0008] Once the binary classifier and the multi-class classifier are trained, a defect classification model is constructed or generated, and a configuration file defines a first combination and a second combination of a neural network architecture and an optimizer, and their parameters. The configuration file further includes the final updated weights of the nodes of the neural network architecture from the binary classifier and the multi-class classifier respectively. As a result, the automatic defect classification model can be used by an automatic inspection device that detects defective products and identifies the defect types of the manufactured parts to be inspected.

[0009] In an effective embodiment of this method, the step of training the binary classifier further includes an initial step of automatically searching for different combinations of a neural network architecture and an optimizer on a search subset of inspection images. The first combination selected for the binary classifier corresponds to the combination that achieves the highest accuracy in identifying defective products from non-defective products at a given number of epochs during the searching step.

[0010] In an effective embodiment of this method, the step of training the multi-class classifier further includes an initial step of automatically searching for different combinations of a neural network architecture and an optimizer using another search subset of inspection images. The second combination of the neural network architecture and the optimizer selected for the multi-class classifier corresponds to the combination that achieves the highest accuracy in identifying different defect types at a given number of epochs during the searching step.

[0011] In an effective embodiment of the present method, the step of training the binary classifier further includes the step of automatically searching for different loss functions and different learning rate schedulers. The first combination is further defined by the loss function and the learning rate scheduler that achieve the highest accuracy together with the neural network architecture and the optimizer when differentiating defective products from non-defective products at a given number of epochs during the search phase. The selection of the loss function and the learning rate is performed automatically. The configuration file of the defect classification model further includes parameters from the loss function and the learning rate scheduler selected for the binary classifier.

[0012] In an effective embodiment of the present method, the step of training the multi-class classifier further includes the step of automatically searching for different loss functions and different learning rate schedulers. The second combination is further defined by the loss function and the learning rate scheduler that achieve the highest accuracy together with the neural network architecture and the optimizer when identifying defect types at a given number of epochs during the search phase. The configuration file of the defect classification model further includes parameters from the loss function and the learning rate scheduler selected for the multi-class classifier.

[0013] In an effective embodiment of the present method, the updated weights and parameters of the selected neural network architecture, optimizer, loss function, and learning rate scheduler are packaged in a configuration file that can be loaded by an automatic inspection device.

[0014] In an effective embodiment of the present method, different neural network architectures include at least one of the following neural network architectures, namely, ResNet34, NesNet50, ResNet101, ResNet152, WideResNet50, WideResNet101, IncptionV3, or InceptionResNet.

[0015] In an effective embodiment of this method, different optimizers include at least one of the Adam optimizer or the SGD optimizer.

[0016] In an effective embodiment of this method, different loss functions include at least one of the cross-entropy loss function or the negative log-likelihood loss function.

[0017] In an effective embodiment of this method, different learning rate schedulers include at least one of the decaying learning rate scheduler or the cyclic learning rate scheduler.

[0018] In an effective embodiment of this method, the automatic inspection device is trained to detect different defect types in at least one of the following products, namely at least one of semiconductor packages, wafers, single-sided PCBs, double-sided PCBs, multi-layer PCBs, or substrates.

[0019] In an effective embodiment of this method, the defect types include one or more of the following: plating penetration, foreign matter inclusion, unfinished products, cracks, stains, abnormal circuits, resist residues, deformations, scratches, clusters, or metal film residues.

[0020] In an effective embodiment of this method, the step of acquiring inspection images includes the step of capturing, via a graphical user interface, the selection of one or more image folders where the inspection images are stored.

[0021] In an effective embodiment of this method, the step of training the binary classifier and the multi-class classifier is initiated in response to input being provided via a graphical user interface.

[0022] In an effective embodiment of this method, the step of training the binary classifier and the multi-class classifier is controlled via a graphical user interface such that the input is captured and the training can be paused, terminated, or resumed.

[0023] In an effective embodiment, the method verifies whether the total number of inspection images is sufficient to start the step of training the binary classifier, and if the total number is sufficient, verifies whether the number of inspection images associated with each defect type is sufficient to start the step of training the multi-class classifier. As a result, the step of training the multi-class classifier is only started for defect types with a sufficient number of inspection images for each defect type.

[0024] In an effective embodiment, the method includes the step of using a data augmentation algorithm to increase the number of inspection images of a given defect type when the number of inspection images associated with the given defect type is insufficient.

[0025] In an effective embodiment, the method includes the step of automatically splitting the inspection images into at least a training dataset and a validation dataset in each of the first subset and the second subset before the steps of training the binary classifier and the multi-class classifier. During training, the training dataset is used to set the initial parameters of the first combination and the second combination of the neural network architecture and the optimizer. The validation dataset is used to verify and further adjust the node weights during the steps of training the binary classifier and the multi-class classifier.

[0026] In an effective embodiment, the method includes the step of automatically splitting the inspection images into a test dataset to confirm the parameters and weights of the first combination and the second combination once the binary classifier and the multi-class classifier are trained.

[0027] In an effective embodiment of the method, the number of inspection images used to train the binary classifier and the multi-class classifier in each learning iteration is dynamically adapted as a function of the available physical resources of the processor executing the training step.

[0028] In an effective embodiment of the present method, the number of inspection images passed through the binary classifier and the multi-class classifier in each iteration is bundled in a predetermined batch size to be tested until an acceptable batch size that can be processed by the processor is reached.

[0029] In an effective embodiment of the present method, the step of training the binary classifier and the multi-class classifier is performed by supplying inspection images to the subsequent batch of classifiers, and the number of inspection images in each batch is dynamically adjusted as a function of the availability of processing resources.

[0030] In an effective embodiment of the present method, the step of acquiring inspection images includes scanning an image server and displaying on a graphical user interface a representation of a folder structure including machine identifiers, customer identifiers, recipe identifiers, and lot identifiers or device identifiers that can be selected by the user.

[0031] In an effective embodiment, the present method includes the step of verifying whether an inspection image is already stored on the learning server before copying the inspection image to the learning server.

[0032] According to another aspect, an automatic inspection system that automatically generates a defect classification model through machine learning is provided, and each model is adapted for the inspection of a specific component type. These different defect classification models can be used to inspect different types of manufactured components such as semiconductor components and / or printed circuit board (PCB) components. The system includes one or more dedicated servers including one or more processors and a data storage device, and the data storage device is stored on the dedicated server. The system further includes an acquisition module that acquires an inspection image of a component captured by an inspection device, and the inspection image is associated with label information indicating whether a given image corresponds to a non-defective product or a defective product, and further indicating the defect type if the inspection image corresponds to a defective product.

[0033] This system further comprises a learning application including a binary classifier that can be trained using a first subset of inspection images to determine whether an inspection image corresponds to a good product or a defective product by repeatedly updating the weights of the nodes of a neural network architecture used for the binary classifier. The binary classifier uses a first combination of a neural network architecture and an optimizer. The learning application further includes a multi-class classifier that can be trained using a second subset of inspection images corresponding to defective products to identify the defect types in the inspection images previously determined by the binary classifier to correspond to defective products. The multi-class classifier uses a second combination of a neural network architecture and an optimizer. The multi-class classifier is trained by repeatedly updating the weights of the nodes of the neural network architecture used for the multi-class classifier.

[0034] The learning application includes an algorithm for generating a defect classification model defined by a configuration file from the trained binary classifier and the trained multi-class classifier. The configuration file includes the parameters of the first and second combinations of the neural network architecture and the optimizer, as well as the updated weights of the nodes of each neural network architecture. As a result, the automatic defect classification model can be used by an automatic inspection device that detects the defects when parts to be inspected are added.

[0035] In an effective embodiment of the present system, the data storage device further stores a search module, a first set of different neural network architectures, and a second set of optimizers. The search module is configured to search for different combinations of neural network architectures and optimizers on a search subset of inspection images for training a binary classifier. The search module is further configured to automatically select, for the binary classifier, a first combination of a neural network architecture and an optimizer that achieves the highest accuracy when distinguishing defective products from non-defective products at a given number of epochs.

[0036] In an effective embodiment of the present system, the search module is further configured to search for different combinations of neural networks and optimizers on a search subset of inspection images for training a multi-class classifier. The search module is further configured to automatically select, for the multi-class classifier, a second combination of a neural network architecture and an optimizer that achieves the highest accuracy when identifying defect types at a given number of epochs.

[0037] In an effective embodiment, the present system includes a graphical user interface that allows a user to select one or more image folders storing inspection images and, in response to input received through itself, to start generating an automatic defect classification model.

[0038] In an effective embodiment, the present system includes a database that stores inspection images of parts captured by an inspection device, indicates whether a given image corresponds to a non-defective product or a defective product, and further stores label information indicating the defect type if the inspection image corresponds to a defective product.

[0039] In an effective embodiment of the present system, the data storage device of one or more dedicated servers verifies whether the total number of inspection images is sufficient to start the step of training the binary classifier and the multi-class classifier, and then copies the images to the database and further stores a preprocessing module for processing the images, such as by using a data augmentation algorithm.

[0040] According to yet another aspect, a non-transitory storage medium is provided. The non-transitory storage medium causes a processor to acquire inspection images of parts imaged by an inspection device, the inspection images being associated with label information indicating whether a given image corresponds to a non-defective product or a defective product, and further indicating a defect type if the inspection image corresponds to a defective product. The non-transitory storage medium causes the processor to train a binary classifier using a first subset of the inspection images to determine whether the inspection images correspond to non-defective products or defective products. The binary classifier uses a first combination of a neural network architecture and an optimizer. The non-transitory storage medium causes the processor to train a multi-class classifier using a second subset of the inspection images corresponding to defective products to identify the defect type in the inspection images previously determined by the binary classifier to correspond to defective products. The multi-class classifier uses a second combination of a neural network architecture and an optimizer. The non-transitory storage medium causes the processor to generate a defect classification model including the configuration settings of the first combination and the second combination of the neural network architecture and the optimizer from the trained binary classifier and multi-class classifier. As a result, the computer-readable instructions stored in the non-transitory storage medium can be used by an automatic inspection device that detects defects when a part to be inspected is added.

[0041] Other features and advantages of the embodiments of the present invention will be better understood by reading the preferred embodiments thereof with reference to the accompanying drawings.

Brief Description of the Drawings

[0042]

Figure 1

[0043]

Figure 2

[0044]

Figure 3

[0045]

Figure 4

[0046]

Figure 5

[0047]

Figure 6

[0048]

Figure 7

[0049] Note that the attached drawings show only typical embodiments of the present invention, and therefore, since the present invention can recognize other equally effective embodiments, it should not be construed as limiting the scope thereof.

Mode for Carrying Out the Invention

[0050] In the following description, the same reference numerals are given to the same features in the drawings. In order not to make the drawings overly complicated, when some elements have already been shown in the preceding drawings, they may not be shown in some of the drawings. It should be understood that since the emphasis in this specification is placed on clearly showing these elements and the interactions between the elements, the elements in the drawings are not necessarily drawn to scale.

[0051] The automatic defect classification system, method, and software application described in the present application relate to two-dimensional shape and / or three-dimensional shape automatic inspection and measurement devices. The applicant has already commercialized various inspection devices such as semiconductor package inspection devices (such as GATS-2128 and GATS-6163), printed circuit board inspection devices (STAR REC and NRFEID), and optical appearance inspection devices (wafer or substrate bump inspection devices). The present system that automatically generates one or more proposed defect classification models can be used together with these. The typical systems and processes described with reference to FIGS. 1 to 6 are particularly suitable for the inspection of semiconductor products and PCB products. However, the proposed systems and methods are for other applications and are only for illustrative purposes, and can be used in other industries that require automatic inspection of parts such as the automotive industry. The proposed defect classification system can also be adapted to different automatic appearance inspection devices other than laser triangulation.

[0052] Regarding semiconductor inspection, existing optical inspection systems often include an offline defect detection stage, and the task of classifying the detected defects into product-specific or client-specific classes is manually performed by human operators. There are also systems equipped with an automatic AI / machine learning (ML) classifier that analyzes the images generated by the inspection camera and assigns the defects to a predetermined class in real time. However, it is difficult to build these systems, and they often require data experts and / or AI experts who can properly adjust the classifier. Moreover, usually pre-defined ML models are used, and depending on the type of defects that need to be detected, these are not necessarily the optimal models.

[0053] According to one aspect of the present invention, an artificial intelligence (AI)-based automatic defect classification system is provided. As described in more detail below, when combined with an automatic defect classification model, the inspection apparatus can achieve higher measurement accuracy, thereby reducing the inspection cost and reducing human error throughout the inspection process.

[0054] The proposed system and method enable the automatic generation of one or more defect classification models for use in an automatic part inspection device. A user, such as a machine operator who has no knowledge of AI or has limited knowledge thereof, can use the proposed system and method to build a new detection-classifier model or update an existing detection-classifier model, regardless of whether the inspection device operates in-line or offline. The proposed system and method can build or update classification models for different product types, such as wafers, individual dies, substrates, or IC packages. As a result, in the proposed system and method, the training of the inspection device for detecting defect types for different products is significantly simplified. In some embodiments, the proposed classification-training system can detect changes in the types of parts and / or defects presented and adjust the defect classification model without user intervention or with limited user intervention. Although a human operator (e.g., a process engineer) may still need to verify the model before it is pushed to an in-line inspection device, the learning process is significantly simplified. There are multiple conditions that trigger the creation of a new classification model or an adjustment to an existing classification model, i. when new images are captured for defects from a deficient defect class (i.e., a class for which there are not enough images to properly tune or configure the classification model), ii. when there is a change in the class label (the label can correspond to a defective or good product, or the type of defect), iii. when a newly inspected product occurs (requiring the construction of a new classification model), iv. when retraining is scheduled, and v. when a drift in the classification model is detected by a quality assurance mechanism.

[0055] In an effective embodiment, in the proposed system and method, the optimal model to be used can be automatically selected from a list of existing machine learning (ML) models, and the adjustment of hyperparameters associated with the model can be realized using a simple grid search method.

[0056] In a preferred embodiment, the proposed system and method further have the advantage of being implemented on a dedicated server. That is, the proposed system and method can be implemented in a closed environment without the need to access an AI cloud-based platform. Therefore, the proposed automatic classification training can be implemented in an isolated environment such as a factory where there is no or limited Internet access.

[0057] The term "processing device" includes computers, nodes, servers, and / or dedicated electronic devices configured and adapted to receive, store, process, and / or transmit data such as labeled images and machine learning models. By way of example only, the "processing device" includes microcontrollers and / or processors such as microprocessors, CPUs, and GPUs. These processors are used in combination with a data storage device, also referred to as a "memory" or "storage medium". The data storage device can store instructions, algorithms, rules, and / or image data to be processed. By way of example only, the storage medium includes volatile or non-volatile / persistent memories such as registers, caches, RAM, flash memory, ROM, etc. It goes without saying that the type of memory is selected according to the desired application, whether it should hold instructions or temporarily store, hold, or update data. A schematic diagram of an architecture that is part of or linked to the automatic inspection device is shown in FIG. 7, and the architecture includes such a processing device and data storage device.

[0058] The "classifier" refers to a machine learning algorithm whose function is to classify or predict the class or label to which data such as digital images belong. A "classifier" is a special type of machine learning model. In some cases, a classifier is a discrete-valued function that assigns class labels to data points. In this application, the data points are derived from digital inspection images. A "binary classifier" predicts, with a given accuracy and confidence, one of two "classes" to which a given data set belongs. In the case of manufacturing part inspection, the classes can be "qualified" or "unqualified". A "multi-class" classifier predicts, with a given accuracy and confidence, one of a plurality of classes to which a given data set belongs.

[0059] The "defect classification model" or "model" also refers to a machine learning model. In this specification, the defect classification model is a combination of trained classifiers, and is used in combination with an optimizer, a loss function, and a learning rate scheduler, and its parameters are also adjusted during the training of the classifier.

[0060] A "neural network architecture", also simply called a "neural network", refers to a specific type of machine learning model (or algorithm) based on a set of connected nodes (also called "artificial neurons" or "perceptrons") structured in layers. The nodes of a given layer are interconnected to the nodes of the adjacent layer, and weights are assigned to the connections between the nodes. A bias represents how much a prediction deviates from the target value. A bias can be seen as the difference between the input of a node and the output of the node. There are various neural network architectures, including convolutional neural networks, recurrent neural networks, etc. More specific examples of neural network architectures include the ResNet architecture and the Inception architecture.

[0061] The "loss function" refers to an algorithmic function that measures how much the prediction made by a model or classifier deviates from the actual value. The smaller the number returned by the loss function, the more accurate the prediction of the classifier.

[0062] The "optimizer" refers to an algorithm that associates the loss function with the parameters of the classifier and updates the weights of the nodes of the classifier in response to the output of the loss function. That is, the optimizer updates the weights of the nodes of the neural network architecture to minimize the loss function.

[0063] The "learning rate scheduler" refers to an algorithm that adjusts the learning rate during the step of training a machine learning classifier by decreasing the learning rate according to a predetermined schedule. The learning rate is a hyperparameter that controls the degree to which the classifier needs to be changed according to the estimation error (by adjusting the weights).

[0064] An "epoch" refers to the number of passes or cycles through which the entire dataset passes through a machine learning model or architecture. An "epoch" means presenting the dataset once to the machine learning algorithm.

[0065] With reference to FIGS. 1 to 7, the proposed system 600 (shown in FIG. 6) will be described. This system generally includes a preprocessing module for adjusting inspection images used to construct or adjust a defect classification model (shown in FIG. 1), and a learning application accessible via a learning application programming interface (API) that creates or constructs a defect classification model based on labeled and processed training images (FIG. 2) by training a binary classifier and a multi-class classifier, and a postprocessing module (FIG. 3) that manages the created classification model and updates the inspection device 606 with the newly created / adjusted classification model.

[0066] An effective embodiment of the system 600 is shown in FIG. 6. The system 600 includes an acquisition module 610 that acquires inspection images captured by an inspection device 606 using either a two-dimensional camera or a three-dimensional camera. The inspection device 606 operates via a server 604 that runs an inspection device application and includes a database or data storage device for storing inspection images. Thus, the inspection images are first stored in the inspection device database 608 and then classified or labeled using a defect classification application 618 with label information indicating whether the part corresponds to a defective part and, if it corresponds to a defective part, label information indicating the defect type. Another computer or server 602 runs a learning application 614 and provides a learning API that can be accessed by the inspection device 606. The server 602 includes one or more processors that execute the learning application 614. The server 602 includes a non-transitory data storage device that stores computer-readable instructions for the application. A search module 612, which enables different combinations of classifiers to be searched, is provided as part of the learning application 614. The system 600 preferably includes its own learning database 616 that stores different classifiers, optimizers, loss functions, and rate schedulers that can be used when building or updating a defect classification model, as well as configuration settings and parameters for these machine learning algorithms. Preprocessing

[0067] Figure 1 is a diagram schematically showing an effective preprocessing module 10, which is part of the proposed system. The preprocessing module typically creates a training dataset used by a learning application. The training dataset generally includes labeled inspection images, i.e., inspection information such as "no defect" or "defect present", or images tagged or labeled with a specific "defect type". At step 104, one or more servers can be scanned to trigger or activate the proposed system, obtain inspection images captured by the camera of the inspection device, and store the inspection images and the label information or class information associated therewith. Classes or labels can be, for example, 0 for non-defective items for n different types of defects, numbers 1 to n for defects due to plating penetration, 2 for defects due to foreign matter inclusion, 3 for unfinished products, 4 for polyimide (PI) cracks, etc. Any number of defect types can be used, such as 5 to 100 different types of defects. Thus, the label can be any alphanumeric indicator used to tag or provide a display of the image content, such as whether the image corresponds to a defective item or a non-defective item, and the type of defect if the inspection image corresponds to a defective item. Usually, most of the inspection images captured by optical inspection correspond to non-defective items as long as there are no problems in the manufacturing process. Thus, most of the inspection images generated by the optical inspection system are labeled or associated with a non-defective or defect-free label (or class). Usually, only a small part of the inspection images, such as 0.01% to 10% as an example, correspond to defective items. In this case, it is necessary to specifically label or classify the inspection images according to the defect type. As shown in Figure 6, one or more servers (reference number 604) for storing inspection images are part of the inspection device (reference number 606 in Figure 6) or are linked to the inspection device.A schematic diagram of the effective architecture of one or more computers or servers 604 is further detailed in FIG. 7. The architecture includes a processing device (e.g., a two-dimensional PC that provides a graphical user interface via an Equipment Front End Module (EFEM), a three-dimensional processing PC, and a three-dimensional GPU-equipped PC, etc.) and a data storage device 608. The two-dimensional camera and the three-dimensional camera capture inspection images using a two-dimensional or three-dimensional frame grabber, and the images are processed by the CPU and / or GPU of the computer or server 604 and then stored in the inspection device database 608. Note that the architectures shown in FIGS. 6 and 7 are merely examples, and other configurations are possible. For example, the server 604 that manages and stores images and the learning server 602 can be combined as a single server that can also correspond to the server of the inspection device. Various functions and applications including image storage and management, training, and component inspection can be executed from one or more servers / computers.

[0068] In the exemplary embodiments shown in FIGS. 1-7, the inspection images are images of semiconductor components or PCB components such as semiconductor packages, silicon wafers or wafers of other materials, single-sided PCBs, double-sided PCBs, multi-layer PCBs, and substrates. Defect types are merely examples and include plating penetration, foreign matter inclusion, unfinished products, cracks, stains, abnormal circuits, resist residues, deformations, scratches, passivation film abnormalities, clusters, or residues of metal films. Since the number and type of defects can vary depending on the type of component being inspected, this list of defects is of course not exhaustive.

[0069] In a typical embodiment, one or more servers 604 store inspection images in folders organized according to a given folder structure at different folder levels such as machine name, customer name, recipe or part and lot. A typical embodiment of the folder structure is shown in connection with FIG. 4, where the folder structure 408 of the server is presented via a graphical user interface (GUI) 400. The GUI shows a folder structure that matches the folder structure of one or more servers, enabling the selection of training images used by a learning application to retrain or create / build a new defect classification model. Thus, the image folder structure or branch structure is preferably scanned periodically as in step 102. The folder structure 408 presented to the user via the GUI may be dynamically updated by the system in step 106 (FIG. 1) to correspond to the latest folder structure and content of the server 604. With the proposed system and method, a new classification model can be constructed and / or an existing classification model can be adjusted, so that while the inspection device is operating (inline), the folder structure presented via the GUI preferably reflects the current state of the image storage server, because new inspection images can be continuously captured to train one or more classification models, while inspection images are selected and imported via the GUI.

[0070] Continuing to refer to FIG. 1, at step 108, a folder containing inspection images can be selected for retraining and / or a new classification model can be created, and the selection is made via the GUI and used by the system to fetch and load the images used for the training step. Thus, as shown in FIG. 4, the system receives via the GUI a selection of one or more folders containing inspection images used for training. In an effective embodiment, the selection may include one or more upper-level folders such as a "part" folder as a means of selecting all lower-level folders, i.e., all "lot" folders.

[0071] The step of training the classification model can be started by input being performed via the GUI, such as by using button 404 in FIG. 4 corresponding to step 112 in FIG. 1. The GUI can further control the learning process by stopping or resuming the training step as needed (button 406 in FIG. 4). In start / initiation step 112, the total number of selected inspection images is preferably calculated or tabulated and displayed on the GUI (see pane 402 in FIG. 4). To retrain or create a classification model, a minimum number of inspection images for training is required. For the same reason, i.e., to perform proper training and / or create defect classifiers for each individual defect type, it is preferable that the system further calculates the number of inspection images selected for each defect type to confirm that the minimum required number of images has been collected. As will be explained in more detail below, a minimum number of images per defect type is required to prevent bias in the classification model. If the minimum required number of images for a given defect has not been reached, it is preferable that the inspection images corresponding to that defect are discarded from the selection before starting to build the classification model. The inspection images associated with the discarded defect type may ultimately be used for training when the minimum required number of images has been reached (as in item (i) on page 4 above). Thus, the preprocessing module verifies whether the total number of inspection images is sufficient to start the step of training a binary classifier (used to detect pass or fail (i.e., defective vs. non-defective)), and if the total is sufficient, verifies whether the number of inspection images associated with each defect type is sufficient to start the step of training a multi-class classifier (used to detect different defect types), such that the step of training the multi-class classifier is only started for defect types for which there are a sufficient number of inspection images.

[0072] Continuing to refer to FIG. 1, for the reasons described above, preferably after step 110 is executed, step 112 is executed. The system can calculate the total number of selected inspection images, and further show the total number of selected inspection images for each defect type, and display the result to the user via the GUI. After confirming that the number of selected inspection images meets the minimum required learning requirements, step 112 may be triggered by the user via an input performed through the GUI, such as using button 402 as shown in FIG. 4. If the number of inspection images for learning is insufficient, a message may be displayed asking the user to newly select inspection images.

[0073] In step 114, the selected inspection images are preferably pre-processed before being transferred and stored in the learning server 602 shown in FIG. 6. The step of pre-processing the images may include the step of extracting relevant information from the inspection images and converting the images according to techniques well-known in the art, such as image trimming, contrast adjustment, histogram equalization, binarization, image normalization and / or image standardization. Note that in a typical embodiment, different servers are used, such as one or more servers 604 associated with the inspection apparatus and the learning server 602 associated with the learning application. For this reason, the inspection images selected for learning are copied and transferred from server 604 to server 602. However, in other embodiments, it is conceivable to use the same server, and its memory is divided to store in-line inspection images and to store the selected learning images.

[0074] Continuing to refer to FIG. 1, the system stores inspection image information at step 116 and checks for checksum verification within a database associated with or part of learning server 602. This verification step can avoid image duplication on the learning server, thereby verifying the uniqueness of each image before copying a new image into the database. Inspection images identified at step 120 that are not yet stored on the learning server are updated or copied onto the learning server according to step 118.

[0075] Preferably, once transferred to learning server 602, the inspection images are split or decomposed into at least a training dataset and a validation dataset. Thus, the system is configured to automatically split the inspection images into at least a training dataset and a validation dataset in each of a first subset and a second subset prior to the step of training a binary classifier and a multi-class classifier. The first subset includes images for training a binary classifier (i.e., the first subset includes images labeled as having defects and images labeled as having no defects), and the second subset will include images of the first subset labeled as defective and further labeled with defect types. Images within the training dataset are used during training to adjust or change the weights of the nodes in different layers of a neural network architecture using an optimizer to reduce or minimize the output of a loss function. The validation dataset is then used to measure the accuracy of the model using the adjusted weights determined during the training of the binary classifier and the multi-class classifier.

[0076] The learning dataset and the validation dataset are selectively used to train and adjust the weights of the classifier nodes. More preferably, the inspection images are split into three datasets: the learning dataset and the validation dataset as described above, and a third "test" dataset or "final validation" dataset used to verify the final state of the classification model after learning. That is, the test dataset is used by the system to confirm the final weights of the neural network architectures of the binary classifier and the multi-class classifier after learning. Training

[0077] FIG. 2 is a diagram schematically showing steps of a learning process executed by a learning application (or a learning module) to automatically build a defect classification model, and each model is adapted to a specific manufacturing process, a component model, or client requirements. The learning application is a software program stored on server 602 (shown in FIG. 6) that includes different sub-modules. The learning module is managed by a state machine that verifies whether a calling action such as an end, initialization, training, pause, success or failure / exception is permitted at a given moment. The learning module includes a learning API that includes programming functions for managing a learning session. The learning API cab functions include, by way of example only, an initialization function, a resume function, a start function, a pause function, an end function, an evaluation function, a getStatus function, a getPerformance function, and a getTrainingPerformance function. The initialization function adjusts each learning cycle by verifying the content of a first data set and a second data set, including, for example, ensuring that all classes have sufficient sample images, i.e., the number of images per class exceeds a given threshold, and that a subset of search, training, validation, and test images for training each classifier has a predetermined size. The initialization module also initializes the defect classification model to be built using the parameters of a previously built model or using predefined or random weights for each classifier. For this purpose, a configuration file including initial parameters of a first combination and a second combination of a neural network architecture and an optimizer is loaded when training is started. The configuration file can take different formats such as, by way of example, the JSON format.The initial configuration file, by way of mere example, may include fields such as a classifier model loaded during training, an optimizer loaded during training including a learning rate decay factor used, a data augmentation algorithm used when class samples are imbalanced, the number of epochs for which stable accuracy needs to be maintained, etc. The start function is what initiates the learning process, and the learning operation is initiated using the parameters of the individual field portions of the initial configuration file. The evaluation function evaluates the trained defect classification model against an evaluation dataset of inspection images and returns an average accuracy expressed as a percentage, i.e., the percentage of predictions made correctly.

[0078] Accordingly, the learning application can be called by the inspection device via the learning API and can be trained to determine whether an inspection image corresponds to a good product or a defective product (represented by steps 208 and 214 on the left side of FIG. 2), and a multi-class classifier (also called the "defect type classifier" and represented by steps 210 and 216 on the right side of FIG. 2) that can be trained to identify the defect type within an inspection image determined to have a defect by the binary classifier, first loads or includes these.

[0079] To "train" a classifier means that the weights of the nodes in the different layers that form the classifier (binary or multi-class) are iteratively adjusted to maximize the accuracy of the classifier's predictions over a given number of trials (or epochs). The optimizer selected in combination with the neural network architecture is used during training to iteratively adjust the node weights. Once trained, the weights associated with the multiple nodes of the classifier are set, defining a classification model that can be used for automatic part inspection.

[0080] Therefore, the inspection images selected to create a new classification model and / or adjust an existing model are split into a first subset (a learning subset, a validation subset, a test subset, and, optionally, a search subset) for training a binary classifier, and the inspection images determined to have a defect form a second subset of inspection images to be used for training a multi-class classifier.

[0081] The proposed system and method are particularly advantageous in that different combinations of neural network algorithms and optimizer algorithms can be used for the binary classifier and the multi-class classifier. Further, as will be described in more detail below, the step of determining the best combination of neural network architecture and optimizer for the binary classifier and the multi-class classifier can be performed through a search phase.

[0082] Accordingly, the binary classifier may use a first combination of a neural network (NN) architecture and an optimizer, while the multi-class classifier may use a second combination of a neural network architecture and an optimizer. Note that the binary classifier can be another type of classifier such as, for example, a decision tree, a support vector machine, or a naive Bayes classifier. Preferably, the first combination and the second combination may further include a selection of a loss function algorithm and a related learning rate coefficient. The first combination and the second combination may or may not be the same, but experiments have shown that generally better results are obtained when the first combination and the second combination of the neural network architecture and the optimizer are different between the binary classifier and the multi-class classifier. As an example, the first combination of the neural network architecture and the optimizer for the binary classifier can be the ResNet34 architecture and the Adam optimizer, while the neural network and the optimizer for the multi-class classifier can be the ResNet152 architecture and the SGD optimizer.

[0083] Continuing to refer to FIG. 2, at step 202, data augmentation may be performed on the inspection images because the number of inspection images associated with the same defect type is insufficient to train the defect classification model or is negligible compared to other defect classes. This step serves to balance the number of images for each defect type to improve learning accuracy and avoid biases that would otherwise occur for defect types with significantly more inspection images compared to other defect types. The data augmentation algorithm applies random transformations to a given training inspection image, thereby creating new images and increasing the number of images for a given class. The transformation can be a spatial transformation (such as rotation or inversion of the image), but by way of example only, other types of transformations including changing the red-green-blue (RGB) values of the pixels are also possible.

[0084] In step 204, the learning API dynamically loads an initial configuration file (or initial learning settings), which may include various learning parameters such as a first combination of a neural network architecture and an optimizer used to train a binary classifier (step 214), and a second combination of a neural network architecture and an optimizer used to train a multi-class classifier (step 216). The configuration file and / or learning settings may further include the presentation of a loss function algorithm to be used for training the binary classifier and the multi-class classifier (which may or may not be different for the two classifiers), and the presentation of a learning rate scheduler algorithm (and coefficients) to be used for training the binary classifier and the multi-class classifier (which may or may not be different for the two classifiers). By way of mere example, different neural network architectures that may be used by the binary classifier and / or the multi-class classifier may include NesNet50, ResNet101, ResNet152, WideResNet50, WideResNet101, IncptionV3, and InceptionResNet. Examples of optimizer algorithms that can be used to train the binary classifier and / or the multi-class classifier include the Adam optimizer and the Stochastic Gradient Descent (SGD) optimizer. Examples of loss function algorithms include the cross-entropy loss function and the negative log-likelihood loss function, and examples of learning rate scheduler algorithms include the decaying learning rate scheduler and the cyclic learning rate scheduler. The initial configuration file may further include the weights of each node of the classifier, if necessary.

[0085] The above examples of neural network architectures, optimizers, loss functions, and learning rate schedulers are not exhaustive, and the present invention may be used with different types of architectures, optimizers, loss functions, and learning rate schedulers. The first combination and the second combination of learning parameters and setting values may further include other types of parameters such as the number of epochs in addition to those described above. Preferably, the configuration file (or learning settings) can be updated to add or remove any number of neural network architectures, optimizers, loss functions, and learning rate schedulers.

[0086] In an effective embodiment, the proposed method and system are further advantageous in that different types of neural network architectures and optimizers can be tried or explored to select the best or more accurate combination of architecture and optimizer for a given product type or manufacturing part type, and then the binary classifier and the multi-class classifier can be fully trained. In other words, the proposed method and system include the step of trying and exploring different combinations of neural network and optimizer (and optionally loss function and learning rate scheduler) for training the binary classifier and for training the multi-class classifier to select the "best" or "optimal" combination for fully training the binary classifier and the multi-class classifier, i.e., the combination that achieves the highest accuracy. Continuing to refer to FIG. 2, after the step of exploring or trying 206 is executed, the step of training 212 is executed. The system, at step 208, tests (or explores / tries) different combinations of neural network and optimizer, and more preferably different combinations of loss function and learning rate scheduler, on a reduced subset of the training test images at a given number of epochs for both the binary classifier and the multi-class classifier. Following the exploration phase, a first combination having the highest classification accuracy is selected for the binary classifier.

[0087] The purpose of the initial step of exploring different combinations of neural networks and optimizers is to identify and select the pair of neural network and optimizer that achieves the highest accuracy for a given subset of inspection images. More specifically, the step of exploring different combinations of neural networks and optimizers may include the step of starting several abbreviated learning sessions using different pairs of neural networks and optimizers, and the step of recording the performance (i.e., accuracy) of each pair tried on the reduced dataset (i.e., the exploration dataset). For example, before constructing a defect classification model using a given combination of a binary classifier and an optimizer, n different pairs of neural networks and optimizers, such as ResNet34 as the neural network and Adam as the optimizer, InceptionResNet as the neural network and stochastic gradient descent as the optimizer, ResNet34 as the neural network and SDG as the optimizer, are tried. The accuracy of each pair is determined using the reduced dataset of validation images, and the pair with the highest accuracy is selected and then used in the training step.

[0088] Similarly, when exploring pairs of neural networks and optimizers, different loss functions and learning rate scheduler coefficients can be tried or explored. Thus, at a given number of epochs, the loss function and learning rate scheduler that, together with the NN architecture and optimizer, achieve the highest accuracy (expressed as the percentage of correctly made predictions relative to the total number of predictions made) in distinguishing defective items from good items on the exploration subset of images are identified and retained for the step of training the binary classifier.

[0089] Similarly, in step 210, different combinations of neural networks and optimizers (and possibly loss functions and learning rate schedulers) are tried (or explored) on a reduced subset of training test images to determine the best combination to use to fully train the multi-class classifier. The exploring step is also performed for a given number of epochs. The combination with the highest classification accuracy is selected for the multi-class classifier.

[0090] As mentioned above, the number of epochs may be a parameter of the training system. Thus, in advantageous embodiments, the exploration and training phase may be automatically stopped once a predetermined number of epochs has been reached for each combination of binary classifier and multi-class classifier. Preferably, the exploration and training step may be controlled, e.g., stopped, restarted, and terminated, by the user via a GUI at any point during the exploration phase.

[0091] In an advantageous embodiment of the present system, the search and train steps can be bypassed by loading a configuration file containing the initial parameters associated with the binary classifier and the initial parameters associated with the multi-class classifier, in which case steps 206, 208, and 210 are bypassed.

[0092] 2, after the search phase is completed or bypassed, the steps of training the binary classifier and the multi-class classifier may begin. Training the binary classifier, in one advantageous embodiment, begins in step 214 using a combination of the neural network algorithm and the optimizer algorithm determined during the search phase of step 208. The test images forming the first image subset are used to train the binary classifier. All of the selected images can form the first subset, or only a portion of the selected images.

[0093] The training of the multi-class classifier preferably starts at step 216 using the combination of neural network and optimizer determined to be the most efficient and / or accurate during the exploration phase in step 210, similar to the binary classifier. In this case, the subset of inspection images used to train the multi-class classifier is composed of subsets of the first subset, i.e., this second subset includes inspection images classified as "with defect" by the binary classifier.

[0094] During the steps of training the binary classifier and the multi-class classifier, the weight of the nodes of the binary classifier and the multi-class classifier is iteratively adjusted based on the parameters of the optimizer, for example, after each epoch, using the training image dataset and the validation image dataset. The adjustment of the neural network parameters includes automatically adjusting the weights applied to the nodes of different layers of the neural network using the selected optimizer, loss function, and learning rate coefficient until the difference between the actual result and the predicted result of each learning path is small enough. Therefore, the validation subset is used as representing the actual result of the predicted component state (no defect or with defect, and the type of defect). The adjustment of the optimizer may include iteratively adjusting the hyperparameters of the optimizer used to control the learning process. Note that the learning process is fully automated and is autonomously executed by providing the initial configuration file to the training API.

[0095] In an effective embodiment of the present system and method, the user may be able to pause the training. When a pause command or a stop command is received, the system saves all information related to the training, such as the current configuration settings and the number of epochs executed, in a database on the learning server. When the training step is resumed, the system takes in all the information in the database as the resumption point. FIG. 5 shows an available GUI that can monitor the state of the defect classification model construction process (window 500) and the current iteration and progress of the training (see the lower section 506 of window 500). The GUI enables the learning process to be further paused (502) or terminated (504) as needed.

[0096] In one advantageous embodiment of the system, during the learning process, the number of test images used to train the binary classifier and the multi-class classifier in each learning iteration is dynamically adapted as a function of the available physical resources of the processor executing the training step. More specifically, the batch size, defined as the number of images passed through the binary classifier and the multi-class classifier in each iteration, can be dynamically changed, as indicated by step 218. Advantageously, this step allows the learning process to be adjusted in real time as a function of the available physical / processing resources for executing the training step. In an advantageous embodiment, the batch size may have a predefined value, and different batch sizes may be tried until the training system detects a warning or indication (such as a memory error) that the processing resources (such as a GPU) are fully utilized. In this case, the next smaller batch size is tried until an acceptable batch size that can be processed by the processor (typically a GPU) is reached. In other words, a subset of the test images fed into the classifier is fed into subsequent batches, with the number of test images in each batch being dynamically adjusted as a function of the availability of processing resources (i.e., available processing power or processor utilization). This feature or option of the training system eliminates the need to know in advance the hardware specifications or learning model requirements or parameter sizes. This feature also allows the training system to be highly portable, as different manufacturing plants may have different server / processing unit requirements and / or specifications.

[0097] The steps of training the binary classifier and the multi-class classifier are completed when the accuracy of the classifier reaches a given accuracy threshold (such as exceeding 95%) at a given number of epochs. Therefore, the defect classification model is constructed from the trained binary classifier and multi-class classifier, and at the end of the learning session, it is defined by a configuration file containing the final updated parameters of the first combination and the second combination of the neural network and the optimizer. The selected neural network, optimizer, loss function, and learning rate scheduler are packaged in a configuration file that can be loaded by the automatic inspection device. The configuration file may include parameters such as the user of the neural network architecture of the binary classifier (e.g., ResNet34), the source model of the binary classifier (including weight settings), the user of the neural network architecture of the multi-class classifier (e.g., InceptionResNet), the source model of the multi-class classifier (including weight settings), the optimizer (e.g., Adam) for the binary classifier and multi-class classifier, the learning rate (e.g., 0.03), and the learning rate decay coefficient (e.g., 1.0). As a result, the automatic defect classification model can be used by the automatic inspection device to detect defective products and identify the defect types of the manufacturing parts to be inspected. Preprocessing

[0098] The post-processing module shown in Figure 3 involves various modules that, once the defect classification model is built, store it (step 304), update the GUI of the inspection device using the learning results (step 306), and / or update the database of the training and / or inspection device using the newly created or updated existing model (308). Preferably, before transferring the build of the defect classification model to the inspection device, the accuracy of the first combination and the second combination of the optimizer and the binary / multi-class classifier is demonstrated using a test dataset.

[0099] Thus, in step 310, a resultant defect classification model is generated, which includes the types and parameters of the binary classifier and the multi-class classifier in its configuration file. For example, a defect classification model for a new semiconductor component may include, in the form of a configuration file, a first combination of a neural network architecture, optimizer, loss function, and learning rate scheduler used for the binary classifier, and the associated parameter settings for each of those algorithms, and a second combination of a neural network architecture, optimizer, loss function, and learning rate scheduler used for the multi-class classifier, and the associated parameter settings for each of those algorithms.

[0100] The defect classification model may be stored in a database located on the training server and / or the server of the inspection device. Results from the training process, such as the selected first and second combinations and the corresponding first and second accuracies, may be displayed in the GUI. In one embodiment, the results may be exported to a performance report (step 312).

[0101] In use, the automatic defect classification application loads the appropriate defect classification model according to the part type selected by the operator via a GUI. Thus, each part type can be associated with its own defect classification model, with each model tuned and trained to optimize its accuracy for a given part type or client requirement. Automatic defect classification can advantageously detect new defects captured by the optical system, such as by classifying new / unknown defects into an "unknown" category or label. If the number of "unknown" defects in a given lot exceeds a given threshold, the application can be configured to generate a warning that the classification model needs to be updated, and in some advantageous embodiments, the proposed system and method can automatically update (or retrain) the classification model.

[0102] The proposed method and system for generating an automatic defect classification model via machine learning for use in an automatic inspection device can be effectively deployed on one or more servers at the customer site (where "customer" is typically a manufacturing company) without the need to upload confidential data to a cloud-based server. Further, the proposed method and system enable users to control them and create a defect classification model without prior AI knowledge. The proposed method and system can also directly interact with inspection images without relying on a complex relational dataset. The learning application can be extended by adding a new neural network architecture, a new optimizer, a loss function, and a learning rate scheduler. The learning application includes a layer resizing function (layer resizing process) that ensures that the number of outputs of the newly added neural network architecture matches the number of outputs passed as an argument. The learning application further includes a forward function that substitutes the tensor passed as an argument into the input layer of the model and aggregates the outputs. A similar process can be executed to add a new optimizer, a loss function, and a learning rate scheduler. Experimental results

[0103] One advantage of the present application is that the binary classifier, the multi-class classifier, and the optimizers used with these classifiers can be tested in different combinations. The exploration detailed above and defined in steps 206, 208, and 210 of FIG. 2 was implemented and tested in the experiment. The results of this experiment are shown in Table 1, which is an excerpt from the original table containing all combinations and related results.

[0104] Table 1 includes the different learning parameters used in each combination, the number of epochs for which the test was run, and the accuracy results when classifying inspection images. In the table, the combinations in bold are those selected by the system as the best combinations in terms of classification accuracy.

[0105] The original inspection image dataset used contained 159,087 images divided into a first dataset of 144,255 images, 80% of which were further divided into a training dataset and 20% of which were further divided into a validation dataset, and a second dataset of 14,832 images comprising a testing dataset. These inspection images were associated with a total of 23 classes, including defect types and pass types.

[0106] The method eliminated three classes and 295 associated images before starting the training step. Each of these three classes did not comply with the minimum required number of test images with label information corresponding to that class, which was set at 120. As a result, the training step was performed on a training dataset of 115,160 images and a validation dataset of 28,800 images.

[0107] Table 1 shows that testing multiple combinations of classifiers is advantageous when selecting a combination of both binary and multi-class classifiers, as there is significant variation between different combinations for both binary and multi-class classifiers. The choice of loss function associated with a particular optimizer also has a significant impact on the accuracy of the combination.

[0108] Separate runs were performed on the same test images by our training system consistently selecting the ResNet34 model and SGD optimizer for the binary classifier, and deeper models such as ResNet152 and InceptionResNet combined with the SGD optimizer for the multi-class classifier, confirming the accuracy of our system in selecting the best combination in terms of classification accuracy for both binary and multi-class classifiers.

[0109] When the number of epochs increases from 5 to 10, the accuracy of the combination of binary classifiers ranges from approximately 77% to 92%, and the accuracy of the combination of multi-class classifiers ranges from approximately 64% to 82%. The influence on the classification accuracy caused by different combinations was further verified.

Table 1

[0110] Needless to say, many changes can be made to the above embodiments without departing from the scope of the present disclosure.

Claims

1. A computer-implemented method for automatically generating a defect classification model using machine learning for use in an automatic inspection apparatus for inspecting semiconductor components and / or printed circuit board (PCB) components, the method comprising: Obtaining an inspection image of a component imaged by the inspection apparatus, the inspection image being associated with label information indicating whether a given image corresponds to a good product or a defective product, and further indicating a defect type if the inspection image corresponds to a defective product; Training a binary classifier using a first subset of the inspection images to determine whether the inspection images correspond to good products or defective products, the binary classifier using a first combination of a neural network architecture and an optimizer, the binary classifier being trained by iteratively updating weights associated with nodes of the binary classifier; Training a multi-class classifier using a second subset of the inspection images corresponding to defective products to identify a defect type in the inspection images previously determined to correspond to defective products by the binary classifier, the multi-class classifier using a second combination of a neural network architecture and an optimizer, the multi-class classifier being trained by iteratively updating weights associated with nodes of the multi-class classifier; Constructing a defect classification model defined by a configuration file including the parameters of the first combination and the second combination of the neural network architecture and the optimizer, and the updated weights of the nodes of each neural network architecture, from the trained binary classifier and the multi-class classifier, such that the automatic defect classification model can be used by the automatic inspection apparatus to detect defective products and identify the defect types of the components to be inspected; The step of training the binary classifier further includes an initial step of automatically exploring different combinations of a neural network architecture and an optimizer on a first subset of the inspection images; A computer-implemented method.

2. A computer-implemented method for automatically generating a defect classification model using machine learning for use in an automatic inspection apparatus for inspecting semiconductor components and / or printed circuit board (PCB) components, the method comprising: Obtaining an inspection image of a component imaged by the inspection apparatus, the inspection image being associated with label information indicating whether a given image corresponds to a good product or a defective product, and further indicating a defect type if the inspection image corresponds to a defective product; Training a binary classifier using a first subset of the inspection images to determine whether the inspection images correspond to good products or defective products, the binary classifier using a first combination of a neural network architecture and an optimizer, the binary classifier being trained by iteratively updating weights associated with nodes of the binary classifier; Training a multi-class classifier using a second subset of the inspection images corresponding to defective products to identify a defect type within the inspection images previously determined to correspond to defective products by the binary classifier, the multi-class classifier using a second combination of a neural network architecture and an optimizer, the multi-class classifier being trained by iteratively updating weights associated with nodes of the multi-class classifier; Constructing a defect classification model defined by a configuration file including the parameters of the first and second combinations of the neural network architecture and the optimizer, and the updated weights of the nodes of each neural network architecture, from the trained binary classifier and the multi-class classifier, such that the automatic defect classification model can be used by the automatic inspection apparatus to detect defective products and identify the defect types of the components to be inspected; The step of training the multi-class classifier further includes an initial step of automatically searching for different combinations of the neural network and the optimizer on another search subset in the second subset of the inspection images; A computer-implemented method.

3. The computer-implemented method according to claim 1, wherein the first combination selected for the binary classifier corresponds to the combination that achieves the highest accuracy when distinguishing non-defective products from defective products at a given number of epochs.

4. The computer-implemented method according to claim 2, wherein the second combination of the neural network architecture and the optimizer corresponds to the combination that achieves the highest accuracy when identifying the different defect types at a given number of epochs.

5. The step of training the binary classifier further includes the step of automatically exploring different loss functions and different learning rate schedulers, and the first combination is further defined by automatically selecting a loss function and a learning rate scheduler that achieve the highest accuracy together with the neural network architecture and the optimizer when determining non-defective products from defective products at the given number of epochs, and the configuration file of the defect classification model further includes parameters from the selected loss function and learning rate scheduler in the binary classifier. The computer-implemented method according to claim 3.

6. The step of training the multi-class classifier further includes the step of automatically exploring the different loss functions and the different learning rate schedulers, and the second combination is further defined by automatically selecting a loss function and a learning rate scheduler that achieve the highest accuracy together with the neural network architecture and the optimizer when identifying the defect type at the given number of epochs, and the configuration file of the defect classification model further includes parameters from the selected loss function and learning rate scheduler in the multi-class classifier. The computer-implemented method according to claim 4.

7. The computer-implemented method according to claim 5 or 6, wherein the updated weights and the parameters of the selected neural network architecture, optimizer, loss function, and learning rate scheduler are packaged in the configuration file that can be loaded by the automatic inspection device.

8. The computer-implemented method according to any one of claims 1 to 7, wherein the different neural network architectures include at least one of ResNet34, NesNet50, ResNet101, ResNet152, WideResNet50, WideResNet101, InceptionV3, or InceptionResNet.

9. The computer-implemented method according to any one of claims 1 to 8, wherein the different optimizers include at least one of an Adam optimizer or an SGD optimizer.

10. The computer-implemented method according to any one of claims 1 to 9, wherein the different loss functions include at least one of a cross-entropy loss function or a negative log-likelihood loss function.

11. The computer-implemented method according to any one of claims 1 to 10, wherein the different learning rate schedulers include at least one of a decaying learning rate scheduler or a cyclic learning rate scheduler.

12. The computer-implemented method according to any one of claims 1 to 11, wherein the components include at least one of a semiconductor package, a wafer, a single-sided PCB, a double-sided PCB, a multi-layer PCB, or a substrate.

13. The computer-implemented method according to any one of claims 1 to 12, wherein the multi-class classifier is trained to detect at least one defect type including at least one of plating penetration, foreign matter inclusion, unfinished product, crack, stain, abnormal circuit, resist residue, deformation, scratch, cluster, or metal film residue.

14. The computer-implemented method according to any one of claims 1 to 13, wherein the step of acquiring the inspection image includes capturing a selection of one or more image folders in which the inspection image is stored via a graphical user interface.

15. The computer-implemented method according to claim 14, wherein the step of training the binary classifier and the multi-class classifier is initiated in response to an input being made via a graphical user interface.

16. The computer-implemented method according to claim 15, wherein the step of training the binary classifier and the multi-class classifier is controlled to capture an input via the graphical user interface to pause, end, or resume the training.

17. The method includes verifying whether the total number of the inspection images is sufficient to start the step of training the binary classifier, and if the total number is sufficient, verifying whether the number of the inspection images associated with each defect type is sufficient to start the step of training the multi-class classifier. As a result, the step of training the multi-class classifier is started only for the defect types with a sufficient number of the inspection images. The computer-implemented method according to any one of claims 1 to 16.

18. The computer-implemented method according to claim 17, including the step of increasing the number of the inspection images of the given defect type using a data augmentation algorithm when the number of the inspection images associated with the given defect type is insufficient.

19. Before the step of training the binary classifier and the multi-class classifier, the method includes automatically splitting the inspection images into at least a training data set and a validation data set in each of the first subset and the second subset, setting initial parameters of the first combination and the second combination of the neural network architecture and the optimizer using the training data set during training, and adjusting the weights of the nodes during the step of training the binary classifier and the multi-class classifier using the validation data set. The computer-implemented method according to any one of claims 1 to 18.

20. The computer-implemented method according to claim 19, further including the step of automatically splitting the inspection images into a test data set to confirm the parameters of the first combination and the second combination and the adjusted weights once the binary classifier and the multi-class classifier are trained.

21. The computer-implemented method according to any one of claims 1 to 20, wherein the number of the inspection images used to train the binary classifier and the multi-class classifier in each learning iteration is dynamically adapted as a function of the available physical resources of the processor that executes the training step.

22. The computer-implemented method according to any one of claims 1 to 21, wherein the number of the inspection images passed through the binary classifier and the multi-class classifier in each iteration is bundled in a predetermined batch size to be tested until an acceptable batch size that can be processed by the processor is reached.

23. The computer-implemented method according to any one of claims 1 to 22, wherein the step of obtaining the inspection images includes scanning an image server and displaying on a graphical user interface a representation of a folder architecture including a machine identifier, a customer identifier, a recipe identifier, and a lot identifier or a device identifier so that a user can select them.

24. The computer-implemented method according to any one of claims 1 to 23, including the step of verifying whether the inspection images are already stored on the learning server before copying the inspection images to the learning server.

25. An automatic inspection system for generating an automatic defect classification model using machine learning to inspect semiconductor components and / or printed circuit board (PCB) components, the system comprising: One or more dedicated servers including one or more processors and a data storage device, the data storage device being stored on the dedicated server; a dedicated server; An acquisition module for acquiring inspection images of the semiconductor components and / or printed circuit board (PCB) components, the inspection images being associated with label information indicating whether a given image corresponds to a non-defective product or a defective product, and further indicating a defect type if the inspection image corresponds to a defective product. A binary classifier that can be trained using a first subset of the inspection images to determine whether the inspection images correspond to good products or defective products by repeatedly updating the weights of its own nodes, wherein the binary classifier uses a first combination of a neural network architecture and an optimizer. A multi-class classifier that can be trained using a second subset of the inspection images corresponding to defective products to identify the defect types in the inspection images previously determined to correspond to defective products by the binary classifier by repeatedly updating the weights of its own nodes, wherein the multi-class classifier uses a second combination of a neural network architecture and an optimizer. A learning application including: The learning application includes an algorithm for generating a defect classification model defined by a configuration file including the parameters of the first combination and the second combination of the neural network architecture and the optimizer, and the updated weights of the nodes of each neural network architecture, from the trained binary classifier and the trained multi-class classifier. The automatic defect classification model can be used by the automatic inspection system for detecting the defects when a component to be inspected is added. The learning application is provided. The data storage device further stores a search module. The search module is configured to search for different combinations of a neural network architecture and an optimizer on a search subset of the inspection images for training the binary classifier. An automatic inspection system. **Claim 26**: An automatic inspection system that generates an automatic defect classification model using machine learning to inspect semiconductor components and / or printed circuit board (PCB) components, the system comprising: One or more dedicated servers including one or more processors and a data storage device, wherein the data storage device is stored on the dedicated server. A dedicated server. An acquisition module that acquires inspection images of the semiconductor component and / or printed circuit board (PCB) component, wherein the inspection images are associated with label information indicating whether a given image corresponds to a non-defective product or a defective product, and further indicating a defect type if the inspection image corresponds to a defective product. A binary classifier that can be trained using a first subset of the inspection images to determine whether the inspection images correspond to non-defective products or defective products by repeatedly updating the weights of its own nodes, wherein the binary classifier uses a first combination of a neural network architecture and an optimizer. A multi-class classifier that can be trained using a second subset of the inspection images corresponding to defective products to identify the defect type in the inspection images previously determined to correspond to defective products by the binary classifier by repeatedly updating the weights of its own nodes, wherein the multi-class classifier uses a second combination of a neural network architecture and an optimizer. A learning application including The learning application includes an algorithm for generating a defect classification model defined by a configuration file including the parameters of the first combination and the second combination of the neural network architecture and the optimizer, and the updated weights of the nodes of each neural network architecture, from the trained binary classifier and the trained multi-class classifier. The automatic defect classification model can be used by the automatic inspection system for detecting the defect when a component to be inspected is added. The learning application is provided. The data storage device further stores a search module. The search module is configured to search for different combinations of a neural network architecture and an optimizer on a search subset of the inspection images for training the multi-class classifier. An automatic inspection system.

27. The automatic inspection system according to claim 25, wherein the search module is further configured to select the first combination of the neural network architecture and the optimizer that achieves the highest accuracy for the binary classifier when distinguishing good products from defective products at a given number of epochs.

28. The automatic inspection system according to claim 26, wherein the search module is further configured to select the second combination of the neural network architecture and the optimizer that achieves the highest accuracy for the multi-class classifier when identifying a defect type at a given number of epochs.

29. For a processor, to obtain an inspection image of a component imaged by the inspection device, the inspection image being associated with label information indicating whether a given image corresponds to a good product or a defective product, and further indicating a defect type if the inspection image corresponds to a defective product; to train a binary classifier using a first subset of the inspection images to determine whether the inspection images correspond to good products or defective products, the binary classifier using a first combination of a neural network architecture and an optimizer; to train a multi-class classifier using a second subset of the inspection images corresponding to defective products to identify the defect type in the inspection images previously determined to correspond to defective products by the binary classifier, the multi-class classifier using a second combination of a neural network architecture and an optimizer, and to generate a defect classification model including the configuration settings of the first combination and the second combination of the neural network architecture and the optimizer from the trained binary classifier and the multi-class classifier, the automatic defect classification model being usable by the automatic inspection system for detecting the defect when a component to be inspected is added; to train the binary classifier to search for different combinations of a neural network architecture and an optimizer on the first subset of the inspection images; A storage medium storing computer-readable instructions in itself.

30. For a processor, Obtain an inspection image of a component captured by the inspection device. The inspection image is associated with label information indicating whether a given image corresponds to a non-defective product or a defective product, and further indicating the defect type if the inspection image corresponds to a defective product. Train a binary classifier using a first subset of the inspection images to determine whether the inspection images correspond to non-defective products or defective products. The binary classifier uses a first combination of a neural network architecture and an optimizer. Train a multi-class classifier using a second subset of the inspection images corresponding to defective products to identify the defect type in the inspection images previously determined to correspond to defective products by the binary classifier. The multi-class classifier uses a second combination of a neural network architecture and an optimizer, and Generate a defect classification model including the configuration settings of the first combination and the second combination of the neural network architecture and the optimizer from the trained binary classifier and the multi-class classifier. The automatic defect classification model can be used by the automatic inspection system that detects the defect when a component to be inspected is added. Train the multi-class classifier to explore different combinations of the neural network architecture and the optimizer on the second subset of the inspection images. A storage medium storing computer-readable instructions in itself.

Images