Method and apparatus for searching hyper parameter

Abstract

A method and apparatus for searching for optimal hyperparameters are provided through the steps of: performing a first pre-training of an AI model according to a set of hyperparameters using a first low-resolution image set; performing a first fine-tuning of an AI model according to a set of hyperparameters using a second low-resolution image set; and determining optimal hyperparameters based on the performance of the AI ​​model that has undergone the first pre-training and the first fine-tuning.

Description

Technology Field

[0001] This description relates to a method and apparatus for exploring hyperparameters for training an AI model. Background Technology

[0002] The training of AI models is performed based on hyperparameters set by the user, and these hyperparameters are not automatically adjusted during the training. Since hyperparameters have a significant impact on the performance and training speed of the model, they need to be set appropriately to optimize the model for the domain and prevent overfitting during training.

[0003] In image recognition or classification, when the domain of the input images changes, AI models can generally be pretrained and fine-tuned. That is, the AI ​​model learns the basic features of the changed domain using a large image set (pretraining), and the pretrained model can be fine-tuned to suit a specific task using a small set of labeled images. The problem to be solved

[0004] One embodiment provides a method for searching for optimal hyperparameters.

[0005] Another embodiment provides a device for determining hyperparameters.

[0006] Another embodiment provides a defect detection system for a semiconductor manufacturing process. means of solving the problem

[0007] According to one embodiment, a method for searching for optimal hyperparameters is provided. The method comprises the steps of: performing a first pre-training of an AI model according to a set of hyperparameters using a first low-resolution image set; performing a first fine-tuning of an AI model according to a set of hyperparameters using a second low-resolution image set; and determining optimal hyperparameters based on the performance of the AI ​​model that has undergone the first pre-training and the first fine-tuning.

[0008] In the above method, the step of performing a first pre-training of an AI model according to hyperparameter sets using a first low-resolution image set may include the step of performing a first pre-training using a first image of a first database and a first low-resolution image having a relatively lower resolution than the first image.

[0009] In the above method, the step of performing a first pre-training using a first image of a first database and a first low-resolution image having a relatively lower resolution than the first image may include the step of calculating a first loss using the first low-resolution image, the step of calculating a second loss using the first image, and the step of updating the encoder of an AI model based on the first loss and the second loss.

[0010] In the above method, the step of calculating a first loss using a first low-resolution image may include the step of inputting an unmasked patch from the first low-resolution image into an encoder according to a mask ratio, the step of inputting a low-resolution mask token to a token output from the encoder into a first decoder of an AI model, and the step of calculating a first loss based on the difference between the low-resolution restored image output from the first decoder and the first low-resolution image.

[0011] In the above method, the step of calculating a second loss using a first image may include the step of adding a high-resolution mask token to a token output from an encoder and inputting it to a second decoder of an AI model, and the step of calculating a second loss based on the difference between the reconstructed image output from the second decoder and the first image.

[0012] In the above method, the first loss is determined based on a comparison between the low-resolution restored image and parts corresponding to the masked patch in the first low-resolution image, and the second loss can be determined based on a comparison between the restored image and parts corresponding to the masked patch in the first image.

[0013] The above method may further include the step of performing a second pre-training of an AI model with optimal hyperparameters using a first high-resolution image set, and the step of performing a second fine-tuning of an AI model with optimal hyperparameters using a second high-resolution image set.

[0014] The above method further includes the step of generating a first low-resolution image set by down-converting an image of a first database and the step of generating a second low-resolution image set by down-converting an image of a second database, wherein the first database stores unlabeled images and the second database stores labeled images.

[0015] According to another embodiment, an apparatus for determining hyperparameters is provided. The apparatus comprises one or more processors and memory, wherein the memory is configured to store instructions that cause one or more processors to perform a process, and the process comprises the steps of performing pre-training and fine-tuning of an AI model based on a low-resolution image set according to each of a plurality of hyperparameter sets, and determining optimal hyperparameters based on the performance of the AI ​​model pre-trained and fine-tuned based on the low-resolution image set.

[0016] In the above device, the step of performing pre-training and fine-tuning of an AI model based on a low-resolution image set according to each of a plurality of hyperparameter sets may include the step of performing pre-training of the AI ​​model using a first image in a first database and a first low-resolution image down-converted from the first image, and the step of performing fine-tuning of the AI ​​model using a second image in a second database and a second low-resolution image down-converted from the second image.

[0017] In the above device, the step of performing pre-training of an AI model using a first image of a first database and a first low-resolution image down-converted from the first image may include the step of calculating a first loss using the first low-resolution image, the step of calculating a second loss using the first image, and the step of updating the encoder of the AI ​​model based on the first loss and the second loss.

[0018] In the above device, the step of calculating a first loss using a first low-resolution image may include the step of inputting an unmasked patch from the first low-resolution image into an encoder according to a mask ratio, the step of inputting a low-resolution mask token to a token output from the encoder into a first decoder of an AI model, and the step of calculating a first loss based on the difference between the low-resolution restored image output from the first decoder and the first low-resolution image.

[0019] In the above device, the step of calculating a second loss using a first image may include the step of adding a high-resolution mask token to a token output from an encoder and inputting it to a second decoder of an AI model, and the step of calculating a second loss based on the difference between the reconstructed image output from the second decoder and the first image.

[0020] In the above device, the first loss is determined based on a comparison between the low-resolution restored image and parts corresponding to the masked patch in the first low-resolution image, and the second loss can be determined based on a comparison between the restored image and parts corresponding to the masked patch in the first image.

[0021] In the above device, the process may further include the step of performing pre-training and fine-tuning based on a high-resolution image set of the AI ​​model using optimal hyperparameters.

[0022] According to another embodiment, a defect detection system for a semiconductor manufacturing process is provided. The defect detection system includes a capturing device for capturing an image of a wafer during the manufacturing process, and a defect detection device for detecting defects in the image using an AI model, wherein the AI ​​model classifies images of a new domain after being trained through pre-training and fine-tuning based on a low-resolution image set and pre-training and fine-tuning based on a high-resolution image set.

[0023] In the above defect detection system, during the training of the AI ​​model, hyperparameters for training the AI ​​model are determined through pre-training and fine-tuning based on a low-resolution image set, and pre-training and fine-tuning based on a high-resolution image set can be performed using the determined hyperparameters.

[0024] In the above defect detection system, the AI ​​model may include a masked vision transformer.

[0025] In the above defect detection system, pre-training based on the low-resolution image set for the AI ​​model can be performed according to the loss calculated through a comparison between the masked area of ​​the image and the positionally corresponding area in the image reconstructed from the image. Effects of the invention

[0026] By using a low-resolution image set to rapidly perform pre-training and fine-tuning of the AI ​​model at a low cost, optimal hyperparameters can be determined more quickly compared to cases where domain adaptation of the AI ​​model is performed using a high-resolution image set. Brief explanation of the drawing

[0027] FIG. 1 shows a learning device for an AI model according to one embodiment. FIG. 2 illustrates a method for training an AI model according to one embodiment. FIGS. 3 and 4 illustrate a pre-training method based on a low-resolution image set of an AI model according to one embodiment. FIG. 5 is a diagram showing a defect detection system for a semiconductor manufacturing process according to one embodiment. FIG. 6 shows the structure of a vision transformer according to one embodiment. FIG. 7 shows a controller for pre-training and fine-tuning an AI model according to one embodiment. Specific details for implementing the invention

[0028] The embodiments of this description are described below with reference to the attached drawings so that those skilled in the art can easily implement them. However, this description may be implemented in various different forms and is not limited to the embodiments described herein. Furthermore, in order to clearly explain this description in the drawings, parts unrelated to the explanation have been omitted, and similar parts throughout the specification are denoted by similar reference numerals.

[0029] In this description, each of the phrases such as “A or B”, “at least one of A and B”, “at least one of A or B”, “A, B or C”, “at least one of A, B and C”, and “at least one of A, B, or C” may include any one of the items listed together in the corresponding phrase, or all possible combinations thereof.

[0030] In this description, when a part is described as "including" a certain component, it means that, unless specifically stated otherwise, it does not exclude other components but may include additional components.

[0031] Expressions written in the singular in this description may be interpreted as singular or plural unless explicit expressions such as "one" or "singular" are used.

[0032] In this description, "and / or" includes each of the mentioned components and all combinations of one or more.

[0033] In this description, terms including ordinal numbers, such as first, second, etc., may be used to describe various components, but said components are not limited by said terms. Such terms are used solely for the purpose of distinguishing one component from another. For example, without departing from the scope of the present disclosure, the first component may be named the second component, and similarly, the second component may be named the first component.

[0034] In the flowchart described herein with reference to the drawings, the order of operations may be changed, multiple operations may be merged or some operations may be divided, and certain operations may not be performed.

[0035] The Artificial Intelligence (AI) model of the present disclosure is a machine learning model that learns at least one task and may be implemented as a computer program executed by a processor. The task learned by the AI ​​model may refer to a problem to be solved through machine learning or a task to be performed through machine learning. The AI ​​model may be implemented as a computer program executed on a computing device, downloaded via a network, or sold in the form of a product. Alternatively, the AI ​​model may be linked with various devices via a network.

[0036] In some embodiments described below, a relatively low-resolution image reduced from the original image may be used in the first training stage of the AI ​​model. The AI ​​model may be, for example, a masked vision transformer that encodes patches of a given low-resolution image, and some of the patches of the image may be masked. The low-resolution image may include unlabeled images and / or various labeled images. Patches of a given low-resolution image (positional embeddings may be added) may be encoded as tokens, and dummy mask tokens (tokens corresponding to the masked patches) may be added to the encoded tokens. The tokens and dummy mask tokens may be decoded (as a group) by the AI ​​model to generate a reconstructed low-resolution image. The AI ​​model may be trained by comparing the reconstructed low-resolution image with the corresponding low-resolution image. The first training stage may be performed to determine an optimal set of hyperparameters. In the second training stage, relatively high-resolution images may be used compared to the aforementioned low-resolution images, and these relatively high-resolution images may include unlabeled images and / or various labeled images. The AI ​​model may perform pre-training and fine-tuning in the second training stage using the optimal set of hyperparameters determined in the first training stage. The images used in the first and second training stages may belong to a domain (or category, region, etc.) that the AI ​​model has not previously learned. Training the AI ​​model to learn a new region may be possible with a relatively small number of images.

[0037] FIG. 1 shows a learning device for an AI model according to one embodiment, and FIG. 2 shows a learning method for an AI model according to one embodiment.

[0038] Referring to FIG. 1, the AI ​​model learning device (10) may include a controller (100), an AI model (200), a first DB (300), and a second DB (400).

[0039] In one embodiment, the controller (100) uses a set of images of relatively small resolution to perform pre-training and fine-tuning of the AI ​​model (200) according to the hyperparameters of the hyperparameter preset and can determine the hyperparameters that are optimal for training for new domain adaptation.

[0040] In one embodiment, when the domain of images requiring identification changes, the controller (100) can adapt the AI ​​model (200) to the changed domain by performing pre-training and fine-tuning on the AI ​​model (200) using images of the new domain. In this case, a general image set of the existing domain and / or an unlabeled image set of the new domain may be used for pre-training, and a labeled image set of the new domain may be used for fine-tuning.

[0041] In one embodiment, the controller (100) can perform pre-training and fine-tuning of the AI ​​model (200) using a relatively low-resolution image set and determine hyperparameters based on the results of pre-training and fine-tuning using the low-resolution image set, thereby reducing the time required to search for optimal hyperparameters and optimizing training for domain adaptation. For example, when the AI ​​model (200) uses the structure of a vision transformer (ViT), if an image set with a resolution reduced to 1 / 4 is used for pre-training, the number of tokens output from the encoder of the vision transformer can be reduced to 1 / 4, and the time required for pre-training can be reduced.

[0042] Subsequently, the controller (100) performs pre-training and fine-tuning of the AI ​​model (200) using the original image set of the low-resolution image set, thereby forming an association between the AI ​​model trained based on the low-resolution image set and the AI ​​model trained based on the original image set, and enabling the optimal hyperparameters discovered using the low-resolution image set to achieve the best performance even when trained using the original image set.

[0043] In one embodiment, the AI ​​model (200) performs pre-training and fine-tuning according to hyperparameters provided by the controller (100) and may report the results of the pre-training and fine-tuning of the AI ​​model (200) corresponding to each hyperparameter to the controller (100). In one embodiment, the AI ​​model (200) may be a vision transformer that detects defects in an input image or a ViT-based masked autoencoder (MAE).

[0044] In one embodiment, the first DB (300) stores a large number of unlabeled images (in units of millions to billions). The image set stored in the first DB (300) can be used for pre-training (training and testing) of the AI ​​model (200).

[0045] In one embodiment, the second DB (400) stores small-scale labeled images of the changed new domain. The image set stored in the second DB (400) can be used for fine-tuning (training and testing) of the AI ​​model (200).

[0046] Referring to FIG. 2, the controller (100) can perform pre-training of the AI ​​model (200) using a first low-resolution image set according to a set of hyperparameters determined from a hyperparameter preset (S110). The controller (100) can generate a first low-resolution image set used for pre-training the AI ​​model (200) by performing a resolution down conversion or downscaling on an unlabeled image set stored in the first DB (300).

[0047] Hyperparameter presets can be predefined by the user, and the controller (100) can determine a set of hyperparameters from the hyperparameter presets. For example, the hyperparameter presets may include a learning rate preset, a mask ratio preset, and a weight decay preset. For example, the learning rate preset may be {1e-5, 5e-5, 1e-6}, the mask ratio preset may be {0.55, 0.65, 0.75}, and the weight decay preset may be {0.1, 0.2, 0.3}. The controller (100) can determine a set of hyperparameters by selecting hyperparameters from each preset. Below, an example is described in which the controller (100) determines the optimal hyperparameters based on the results of pre-training of the AI ​​model (200) according to three different learning rates.

[0048] The learning rate is a parameter that determines the learning speed of the AI ​​model (200), and in gradient descent, the weight update and the learning rate have a relationship as shown in Equation 1 below.

[0049]

[0050] Referring to Equation 1, the weight at time t+1 is the weight of time point t learning rate and the gradient of the loss function It can be calculated by subtracting the product of. That is, if a learning rate that is too large is applied, the weights of the AI ​​model (200) are updated by too large a size, which increases the likelihood of overshooting the optimal value. Therefore, the loss function is likely to diverge rather than converge. Conversely, if a learning rate that is too small is applied, the weights of the AI ​​model (200) are updated by too small a size, which slows down the learning speed and may take a long time to reach the optimal value. Accordingly, a controller (100) according to one embodiment can determine hyperparameters that can optimize learning for adapting to a new domain quickly and accurately by performing pre-training and fine-tuning of the AI ​​model (200) using a set of images of relatively low resolution.

[0051] Referring to FIG. 2, the controller (100) can perform fine-tuning of the AI ​​model (200) using a second low-resolution image set according to sets of hyperparameters determined by the controller (100) based on weights updated through prior training of the AI ​​model (200) (S120). The controller (100) can generate a second low-resolution image set used for fine-tuning the AI ​​model (200) by performing a resolution down-conversion on a labeled image set stored in the second DB (400).

[0052] Referring to FIG. 2, the controller (100) can determine optimal hyperparameters based on the performance of a pre-trained and finely tuned AI model (200) based on low-resolution image sets (a first low-resolution image set and a second low-resolution image set) (S130).

[0053] Table 1 below shows the performance of an AI model (200) based on pre-training and fine-tuning of a low-resolution image set performed with different learning rates.

[0054]

[0055] In the embodiment of Table 1, the controller (100) can perform pre-training and fine-tuning based on a low-resolution image set (input image with a resolution of 112×112) by applying different hyperparameters, namely learning rates of 5e-5, 1e-5, and 5e-6, respectively, and measure the performance of the AI ​​model (200). In Table 1, the performance measurement of the AI ​​model (200) was performed by measuring Top1 accuracy. Referring to Table 1, the time required for pre-training and fine-tuning of the AI ​​model (200) at each learning rate is roughly the same, but the Top1 accuracy was highest at a learning rate of 1e-5. Therefore, the controller (100) can determine the optimal learning rate of the AI ​​model (200) as 1e-5, and the optimal learning rate can subsequently be used for pre-training and fine-tuning based on a high-resolution image set of the AI ​​model (200).

[0056] Additionally, the controller (100) can determine other hyperparameters, such as the masking ratio and weight attenuation, in the same way. For example, the controller (100) can determine the optimal masking ratio by changing the masking ratio to 0.55, 0.65, and 0.75, performing pre-training and fine-tuning based on a low-resolution image set of the AI ​​model (200), respectively, and then comparing the measured performance of the AI ​​model (200). The controller (100) can determine the optimal weight attenuation size by changing the weight attenuation size to 0.1, 0.2, and 0.3, performing pre-training and fine-tuning based on a low-resolution image set of the AI ​​model (200), respectively, and then comparing the measured performance of the AI ​​model (200).

[0057] Referring to FIG. 2, the controller (100) can perform pre-training and fine-tuning of the AI ​​model (200) using high-resolution image sets according to optimal hyperparameters determined through pre-training and fine-tuning based on low-resolution image sets (S140).

[0058] Here, the optimal hyperparameters determined through pre-training and fine-tuning based on a low-resolution image set are used as is in the pre-training and fine-tuning based on a high-resolution image set of the AI ​​model (200). That is, the controller (100) performs pre-training and fine-tuning based on a high-resolution image set without changing the hyperparameters, so that the optimal hyperparameters determined in the pre-training and fine-tuning based on a low-resolution image set can operate smoothly in the pre-training and fine-tuning based on a high-resolution image set as well.

[0059] For example, the controller (100) can perform high-resolution image set-based pre-training of the AI ​​model (200) using the original image set stored in the first DB (300), and can perform high-resolution image set-based fine-tuning of the AI ​​model (200) using the original image set stored in the second DB (400) and using the weights determined in the high-resolution image set-based pre-training. This is because the original image sets stored in each DB are relatively higher resolution than the low-resolution image sets down-converted from them.

[0060] Table 2 shows the performance of the AI ​​model (200) after the AI ​​model (200) has undergone high-resolution image set-based pre-training and fine-tuning at each learning rate to verify the optimal hyperparameters determined in the low-resolution image set-based pre-training and fine-tuning.

[0061]

[0062] Referring to Table 2, even when an input image with a resolution of 224×224 (relatively high resolution compared to Table 1) is used for pre-training, the performance of the AI ​​model (200) is best when the learning rate is 1e-5. That is, the optimal hyperparameters determined in the pre-training and fine-tuning based on the low-resolution image set of the AI ​​model (200) can be verified to be optimal for pre-training and fine-tuning based on the high-resolution image set as well. Subsequently, inference on images of a new domain can be performed using the AI ​​model (200) that has been pre-trained and fine-tuned based on the high-resolution image set (S150).

[0063] As described above, the controller (100) can determine optimal hyperparameters quickly compared to domain adaptation of an AI model based on a high-resolution image set by using a low-resolution image set to quickly perform pre-training and fine-tuning of the AI ​​model at a low cost.

[0064] FIGS. 3 and 4 illustrate a pre-training method based on a low-resolution image set of an AI model according to one embodiment.

[0065] Referring to FIG. 3, the controller (100) can generate first low-resolution images by performing a resolution down-conversion on the first image of the first DB (300) for pre-training based on a low-resolution image set (S111). Referring to FIG. 4, in one embodiment, the controller (100) can generate first low-resolution images by down-converting the resolution of the first image of the first DB to 1 / 4.

[0066] Referring to FIG. 3, the controller (100) can divide a first low-resolution image into a plurality of patches and input unmasked patches from the first low-resolution image according to the mask ratio to the encoder (210) of the AI ​​model (200) (S112). In some embodiments, the controller (100) can convert the unmasked patches into an embedding vector and add a positional embedding representing positional information of the unmasked patches to the embedding vector.

[0067] Referring to FIG. 4, in one embodiment, the mask ratio is 75%, and 25% of the unmasked patches may be input to the encoder (210). The encoder (210) may output a token corresponding to 25% of the unmasked patches. Referring to FIG. 4, the token output from the encoder (210) may contain or reflect information that the unmasked patch is the patch at position (1, 2) of the total four patches.

[0068] Referring to FIG. 3, the controller (100) can input a low-resolution mask token to a token output from the encoder (210) and input it to a low-resolution decoder (220) (S113). The low-resolution mask token may be a type of dummy added to the token so that the low-resolution decoder (220) can restore an image of the same size as the original image (the first low-resolution image).

[0069] Referring to FIG. 4, the decoder (220) can output a low-resolution restored image from the token of the encoder (210) to which a low-resolution mask token is added. The decoder (220) can reconstruct the entire restored image including the masked patch based on the position information embeddings inherent in the token.

[0070] Subsequently, the controller (100) calculates a first loss based on the difference between the low-resolution restored image output from the low-resolution decoder (220) and the first low-resolution image (S114). The first loss L LR It can be calculated as shown in mathematical formula 2 below.

[0071]

[0072] H in mathematical equation 2 LR represents the number of patches in the height direction of the image, and W LR represents the number of patches in the width direction of the image. Referring to Equation 2, the first loss L LR is the patch at position (i,j) of the first low-resolution image and patch at position (i,j) of the low-resolution restored image It can be calculated based on the difference between them.

[0073] In one embodiment, the first loss may be determined based on a comparison between the low-resolution restored image and the parts corresponding to the masked patch in the first low-resolution image. That is, the loss may be determined based on the difference between the part corresponding to the masked patch restored by the low-resolution decoder (220) from the token of the unmasked patch and the part corresponding to the masked patch of the first low-resolution image.

[0074] Referring to FIG. 3, the controller (100) can input a high-resolution (or original resolution) mask token to a token output from the encoder (210) and input it to a high-resolution decoder (230) (S115). The high-resolution mask token may be a type of dummy added to the token so that the high-resolution decoder (230) can restore an image of the same size as the original image (first image).

[0075] Referring to FIG. 4, the high-resolution decoder (230) can output a high-resolution reconstructed image from a token of the encoder (210) to which a high-resolution mask token is added. The high-resolution decoder (230) can reconstruct the entire reconstructed image including the masked patch based on the positional information embeddings inherent in the token. The high-resolution decoder (230) can recognize the positional information embeddings inherent in the token by upscaling them to high resolution. For example, the positional information (1,2) of the patch can be upscaled to (1,3), (1,4), (2,3), and (2,4).

[0076] Subsequently, the controller (100) calculates a second loss based on the difference between the high-resolution restored image output from the high-resolution decoder (230) and the first image (S116). The second loss L HR It can be calculated as shown in mathematical formula 3 below.

[0077]

[0078] H in mathematical equation 3 HR represents the number of patches in the height direction of the image, and W HR represents the number of patches in the width direction of the image. Referring to Equation 3, the second loss L HR is the patch at position (i,j) of the first image and patch at position (i,j) of the high-resolution restored image It can be calculated based on the difference between them.

[0079] In one embodiment, the second loss may be determined based on a comparison between the high-resolution restored image and the parts corresponding to the masked patch in the first image. That is, the loss may be determined based on the difference between the part corresponding to the masked patch restored by the high-resolution decoder (230) from the token of the unmasked patch and the part corresponding to the masked patch of the original image (the first image).

[0080] Referring to FIG. 3, the controller (100) can update the encoder (210) of the AI ​​model (200) based on the final loss (S117). The final loss L can be calculated as shown in Equation 4 below.

[0081]

[0082] Subsequently, the controller (100) may determine whether the loss function converges or whether a predetermined learning condition is satisfied, and thus the termination of the pre-training of the AI ​​model (200) (S118). When the pre-training of the AI ​​model (200) based on a low-resolution image set is terminated, the controller (100) may perform fine-tuning based on the low-resolution image set of the AI ​​model (200).

[0083] FIG. 5 is a diagram showing a defect detection system for a semiconductor manufacturing process according to one embodiment.

[0084] Referring to FIG. 5, a defect detection system (500) for a semiconductor manufacturing process may include a shooting device (510) and a defect detection device (520).

[0085] When an in-fab wafer is processed by each process equipment (equipment 1, equipment 2, etc.) according to a process sequence (step 1, step 2, ..., step n-1, step n), a photographing device (510) during the process sequence can take an image of the wafer. The photographing device (510) may include a scanning electron microscope (SEM), an optical microscope (OM), a transmission electron microscope (TEM), X-ray inspection equipment, etc.

[0086] A defect detection device (520) according to one embodiment can detect defects in an input image using an AI model (200) when an image transmitted from a shooting device (510) is input.

[0087] An AI model (200) according to one embodiment may be pre-trained and fine-tuned to classify images (images of a new domain) generated from a new product or a new process, etc. Images of a new domain may include a large number of unlabeled images and a small number of labeled images. As described above, the controller (100) may perform pre-training of the AI ​​model (200) using a large number of unlabeled new domain images and perform fine-tuning of the AI ​​model (200) using a small number of labeled new domain images. Additionally, the controller (100) may determine optimal hyperparameters by performing pre-training and fine-tuning based on a low-resolution image set and perform pre-training and fine-tuning based on a high-resolution image set using the determined optimal hyperparameters.

[0088] Optimal hyperparameters are determined through pre-training and fine-tuning based on a low-resolution image set, and when pre-training and fine-tuning based on a high-resolution image set of the AI ​​model (200) according to the determined optimal hyperparameters are completed, the defect detection device (520) can classify images of a new domain using the AI ​​model (200) that has completed pre-training and fine-tuning.

[0089] FIG. 6 shows the structure of a vision transformer according to one embodiment.

[0090] Referring to FIG. 6, an AI model (200) according to one embodiment may include the structure (600) of a vision transformer.

[0091] The controller (100) may divide the input image into multiple patches to determine the class of the input image and generate a token corresponding to each patch. To generate a token from the patches, each patch may be converted into a vector form (flattened), and a fixed-dimensional embedding vector may be generated through a linear transformation. Additionally, positional embedding may be added to the embedding vector so that the relative positions between the patches can be reflected.

[0092] Subsequently, input features can be learned using the transformer's encoder. In the transformer's encoder, Multi-Head Self-Attention is applied, and attention operations can be performed to learn the relationships between different patches.

[0093] Subsequently, the attention output is passed through an MLP (Feedforward Network, FFN) to perform additional non-linear transformations, and layer normalization (LayerNorm) and residual connections are used to stabilize the learning process.

[0094] Subsequently, the loss is calculated (typically Cross Entropy Loss is used), and the weights can be updated by backpropagating the gradient of the loss function.

[0095] FIG. 7 shows a controller for pre-training and fine-tuning an AI model according to one embodiment.

[0096] A controller for pre-training and fine-tuning an AI model according to one embodiment may be implemented as a computer system, for example, a computer-readable medium. Referring to FIG. 7, the computer system (700) includes at least one processor (710) and a memory (720). The memory (720) may be connected to the processor (710) and may store various information for driving the processor (710) or at least one program executed by the processor (710). Alternatively, the memory (720) may be configured to store instructions configured to enable the processor (710) to perform the function, process, or method described in the embodiment.

[0097] The processor (710) may implement the function, process, or method proposed in the embodiment. The operation of the computer system (700) according to the embodiment may be implemented by the processor (710). At least one processor (710) may include at least one of a GPU, a CPU, and an NPU. When the operation of the computer system (700) is implemented by at least one processor (710), each task may be divided among at least one processor (710) according to the load. For example, when one processor is a CPU, the other processor may be any one of a GPU, an NPU, an FPGA, or a DSP.

[0098] In the embodiment of the present description, the memory (720) may be located inside or outside the processor, and the memory may be connected to the processor through various known means. The memory is a volatile or non-volatile storage medium of various forms, and for example, the memory may include read-only memory (ROM) or random access memory (RAM).

[0099] Meanwhile, the embodiments are not implemented solely through the devices and / or methods described so far, but may also be implemented through a program that realizes a function corresponding to the configuration of the embodiments or a recording medium on which such a program is recorded. Such implementation can be easily achieved by a person skilled in the art to which this description pertains, based on the description of the embodiments described above. Specifically, the method according to the embodiments (e.g., an image preprocessing method, etc.) may be implemented in the form of program instructions that can be executed through various computer means and may be recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., either individually or in combination. The program instructions recorded on the computer-readable medium may be specially designed and configured for the embodiments, or they may be known and available to a person skilled in the art of computer software. The computer-readable recording medium may include a hardware device configured to store and execute program instructions. For example, computer-readable recording media may be magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; ROM; RAM; flash memory; etc. Program instructions may include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer through an interpreter, etc.

[0100] Although the embodiments have been described in detail above, the scope of the rights described herein is not limited thereto, and various modifications and improvements by those skilled in the art using the basic concepts defined in the following claims also fall within the scope of the rights described herein.

Claims

Claim 1 A method for searching for optimal hyperparameters, comprising the steps of: performing a first pre-training of an artificial intelligence (AI) model according to a set of hyperparameters using a first low-resolution image set; performing a first fine-tuning of the AI ​​model according to the set of hyperparameters using a second low-resolution image set; and determining the optimal hyperparameters based on the performance of the AI ​​model after the first pre-training and the first fine-tuning. Claim 2 A method according to claim 1, wherein the step of performing a first pre-training of an AI model according to hyperparameter sets using the first low-resolution image set comprises the step of performing the first pre-training using a first image of a first database and a first low-resolution image having a relatively lower resolution than the first image. Claim 3 In claim 2, the step of performing the first pre-training using a first image of the first database and a first low-resolution image having a relatively lower resolution than the first image comprises the step of calculating a first loss using the first low-resolution image, the step of calculating a second loss using the first image, and the step of updating the encoder of the AI ​​model based on the first loss and the second loss. Claim 4 In claim 3, the step of calculating a first loss using the first low-resolution image comprises: inputting an unmasked patch from the first low-resolution image to the encoder according to a mask ratio; inputting a low-resolution mask token to a token output from the encoder and inputting it to a first decoder of the AI ​​model; and calculating the first loss based on the difference between the low-resolution restored image output from the first decoder and the first low-resolution image. Claim 5 In claim 4, the step of inputting an unmasked patch in the first low-resolution image to the encoder according to the mask ratio comprises the step of converting the unmasked patch into an embedding vector, and the step of adding a position information embedding representing the position information of the unmasked patch to the embedding vector and inputting the embedding vector with the added position information embedding to the encoder. Claim 6 In claim 4, the step of calculating a second loss using the first image comprises: adding a high-resolution mask token to a token output from the encoder and inputting it to a second decoder of the AI ​​model; and calculating the second loss based on the difference between the reconstructed image output from the second decoder and the first image. Claim 7 A method according to claim 6, wherein the first loss is determined based on a comparison between the low-resolution restored image and parts corresponding to the masked patch in the first low-resolution image, and the second loss is determined based on a comparison between the restored image and parts corresponding to the masked patch in the first image. Claim 8 A method according to claim 1, further comprising the steps of: performing a second pre-training of the AI ​​model with the optimal hyperparameters using a first high-resolution image set; and performing a second fine-tuning of the AI ​​model with the optimal hyperparameters using a second high-resolution image set. Claim 9 A method according to claim 1, further comprising the step of generating a first low-resolution image set by down-converting an image of a first database, and generating a second low-resolution image set by down-converting an image of a second database, wherein the first database stores unlabeled images and the second database stores labeled images. Claim 10 A device for determining hyperparameters, comprising one or more processors and memory, wherein the memory is configured to store instructions that cause the one or more processors to perform a process, and the process comprises the steps of: performing pre-training and fine-tuning of an artificial intelligence (AI) model based on a low-resolution image set according to each of a plurality of hyperparameter sets; and determining optimal hyperparameters based on the performance of the AI ​​model pre-trained and fine-tuned based on the low-resolution image set. Claim 11 In claim 10, the step of performing pre-training and fine-tuning of an AI model based on a low-resolution image set according to each of the plurality of hyperparameter sets comprises: performing the pre-training of the AI ​​model using a first image in a first database and a first low-resolution image down-converted from the first image; and performing the fine-tuning of the AI ​​model using a second image in a second database and a second low-resolution image down-converted from the second image. Claim 12 In claim 11, the step of performing the pre-training of the AI ​​model using a first image of the first database and a first low-resolution image down-converted from the first image comprises the step of calculating a first loss using the first low-resolution image, the step of calculating a second loss using the first image, and the step of updating the encoder of the AI ​​model based on the first loss and the second loss. Claim 13 In claim 12, the step of calculating a first loss using the first low-resolution image comprises: inputting an unmasked patch from the first low-resolution image to the encoder according to a mask ratio; inputting a low-resolution mask token to a token output from the encoder to the first decoder of the AI ​​model; and calculating the first loss based on the difference between the low-resolution restored image output from the first decoder and the first low-resolution image. Claim 14 In claim 13, the step of calculating a second loss using the first image comprises: adding a high-resolution mask token to a token output from the encoder and inputting it to a second decoder of the AI ​​model; and calculating the second loss based on the difference between the restored image output from the second decoder and the first image. Claim 15 An apparatus according to claim 14, wherein the first loss is determined based on a comparison between the low-resolution restored image and parts corresponding to a masked patch in the first low-resolution image, and the second loss is determined based on a comparison between the restored image and parts corresponding to a masked patch in the first image. Claim 16 A device according to claim 10, further comprising the step of performing high-resolution image set-based pre-training and fine-tuning of the AI ​​model using the optimal hyperparameters. Claim 17 A defect detection system for a semiconductor manufacturing process, comprising a capturing device for capturing an image of a wafer during the manufacturing process, and a defect detection device for detecting defects in the image using an AI model, wherein the AI ​​model classifies images of a new domain after being trained through pre-training and fine-tuning based on a low-resolution image set and pre-training and fine-tuning based on a high-resolution image set. Claim 18 A system according to claim 17, wherein, in the training of the AI ​​model, hyperparameters for training the AI ​​model are determined through prior training and fine-tuning based on the low-resolution image set, and prior training and fine-tuning based on the high-resolution image set are performed using the determined hyperparameters. Claim 19 In paragraph 17, the above AI model is a system comprising a masked vision transformer. Claim 20 A system in which pre-training based on the low-resolution image set for the AI ​​model is performed according to a loss calculated through a comparison between a masked area of ​​the image and a positionally corresponding area in an image reconstructed from the image.

Images