Model training method, system, electronic device and storage medium for semiconductor wafer defect detection

Abstract

The present invention provides a model training method, system, electronic device and storage medium for semiconductor wafer defect detection. The method comprises the following steps: determining the probability of each defect category being sampled based on a training data set, determining a training data stream from the training data set according to the probability of the defect category being sampled, and performing model training using the training data stream to obtain a defect detection model. The model training is performed by sampling the training data stream from the training data set according to the calculated probability of the defect category being sampled, thereby alleviating the problem of imbalanced training data and improving the detection accuracy of the model for the defect category of semiconductor wafers.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a model training method, system, electronic equipment and storage medium for semiconductor wafer defect detection. Background Art

[0002] The production process of semiconductor wafers is very complicated, and the resulting semiconductor wafers have many defect categories, and the characteristics of different defect categories are not very clear. This leads to poor results when quality inspectors conduct manual re-evaluation. Therefore, deep learning-based methods are usually used to automatically identify defect categories.

[0003] However, the defects generated in the actual production process of semiconductor wafers are often concentrated in a limited number of defect categories. If the deep learning model is trained according to the data generated in the actual production process of semiconductor wafers, the learning effect of the trained model for different defect categories will vary greatly, which will affect the model's detection results for semiconductor wafer defect categories. Summary of the invention

[0004] In view of this, an embodiment of the present invention provides a model training method, system, electronic device and storage medium for semiconductor wafer defect detection to solve the problem that the existing model training method affects the model's detection results for the defect categories of semiconductor wafers.

[0005] To achieve the above objectives, the embodiments of the present invention provide the following technical solutions:

[0006] A first aspect of an embodiment of the present invention discloses a model training method for semiconductor wafer defect detection, the method comprising:

[0007] Determine the probability of each defect category being sampled based on a training data set, wherein the training data set includes a plurality of wafer defect images of the defect category;

[0008] A training data stream is determined from the training data set according to the probability of the defect category being sampled, and the training data stream is used to perform model training to obtain a defect detection model, where the defect detection model is used to detect defect information of a semiconductor wafer.

[0009] Preferably, determining the probability of each defect category being sampled based on the training data set includes:

[0010] Based on the training data set, a first probability vector and a second probability vector are determined, wherein the first probability vector and the second probability vector contain the probabilities of each of the defect categories being sampled, the probabilities of at least two of the defect categories being sampled in the first probability vector are different, and the probabilities of each of the defect categories being sampled in the second probability vector are the same.

[0011] Preferably, determining the probability of each defect category being sampled based on the training data set includes:

[0012] use Determine the probability of each defect category being sampled in the training data set to obtain a third probability vector, wherein the third probability vector contains the probability of each defect category being sampled, C is the number of defect categories in the training data set, and p j is the probability of the jth defect category being sampled in the training data set, n j is the number of wafer defect images in the jth defect category, n i is the number of wafer defect images in the i-th defect category.

[0013] Preferably, determining a training data stream from the training data set according to the probability of the defect category being sampled, and using the training data stream to perform model training to obtain a defect detection model includes:

[0014] Determining a first training data stream from the training data set according to the first probability vector;

[0015] Training a first model to obtain a second model according to the first training data stream and a preset number of training rounds, wherein the first model is a deep learning model;

[0016] determining a second training data stream from the training data set according to the second probability vector;

[0017] The second model is trained using the second training data stream to obtain a defect detection model.

[0018] Preferably, determining a training data stream from the training data set according to the probability of the defect category being sampled, and using the training data stream to perform model training to obtain a defect detection model includes:

[0019] Determining a third training data stream from the training data set according to the third probability vector;

[0020] The deep learning model is trained using the third training data stream to obtain a defect detection model.

[0021] Preferably, determining a training data stream from the training data set according to the probability of the defect category being sampled, and using the training data stream to perform model training to obtain a defect detection model includes:

[0022] In the process of performing the tth round of iterative training on the deep learning model, based on the first probability vector, the second probability vector, the number of iterative training rounds t, and the preset total number of iterative rounds, a fourth probability vector is determined, wherein the fourth probability vector includes the probability of each of the defect categories being sampled;

[0023] Determining a fourth training data stream from the training data set according to the fourth probability vector;

[0024] The deep learning model is iteratively trained according to the fourth training data stream until the number of iterative training rounds t is greater than the total number of iterative rounds to obtain a defect detection model.

[0025] Preferably, determining the first probability vector and the second probability vector based on the training data set includes:

[0026] Determine a first probability vector based on the number of defect categories in the training data set, the number of wafer defect images in each defect category, and a first value of a preset parameter;

[0027] A second probability vector is determined based on the number of defect categories in the training data set, the number of wafer defect images in each defect category, and the second value of the preset parameter.

[0028] Preferably, using the second training data stream to train the second model to obtain a defect detection model includes:

[0029] Freeze the backbone network layer of the second model;

[0030] Randomly initializing the training parameters of the fully connected layer of the second model with the backbone network layer frozen;

[0031] The second model with the backbone network layer frozen and the fully connected layer randomly initialized is trained using the second training data stream until the second model converges to obtain a defect detection model.

[0032] Preferably, using the second training data stream to train the second model to obtain a defect detection model includes:

[0033] Freeze all training parameters of the second model;

[0034] Scaling the training parameters of the fully connected layer of the second model with all training parameters frozen;

[0035] Based on the second training data stream, the second model with all training parameters frozen and training parameters of the fully connected layer scaled is trained until the second model converges to obtain a defect detection model.

[0036] Preferably, using the second training data stream to train the second model to obtain a defect detection model includes:

[0037] Normalizing the training parameters of the fully connected layer of the second model;

[0038] Based on the second training data stream, the second model with normalized training parameters of the fully connected layer is trained until the second model converges to obtain a defect detection model.

[0039] Preferably, in the process of performing the tth round of iterative training on the deep learning model, determining the fourth probability vector based on the first probability vector, the second probability vector, the number of iterative training rounds t, and the preset total number of iterative rounds includes:

[0040] During the tth round of iterative training of the deep learning model, , determine the fourth probability vector, where T is the preset total number of iterations, is the probability of defect category j being sampled in the first probability vector, is the probability of defect category j being sampled in the second probability vector, is the probability of defect category j being sampled in the fourth probability vector calculated during the t-th round of iterative training.

[0041] Preferably, before determining the probability of each defect category being sampled based on the training data set, the method further includes:

[0042] Set the zoom range, color transformation amplitude, and translation amplitude of the wafer defect images in the training dataset.

[0043] A second aspect of an embodiment of the present invention discloses a model training system for semiconductor wafer defect detection, the system comprising:

[0044] A determination unit, configured to determine the probability of each defect category being sampled based on a training data set, wherein the training data set includes a plurality of wafer defect images of the defect category;

[0045] A training unit is used to determine a training data stream from the training data set according to the probability of the defect category being sampled, and use the training data stream to perform model training to obtain a defect detection model, wherein the defect detection model is used to detect defect information of a semiconductor wafer.

[0046] A third aspect of an embodiment of the present invention discloses an electronic device, including a memory and a processor;

[0047] The memory is used to store computer programs;

[0048] The processor is used to execute the computer program, specifically to implement the model training method for semiconductor wafer defect detection disclosed in the first aspect of an embodiment of the present invention.

[0049] A fourth aspect of an embodiment of the present invention discloses a computer storage medium for storing a computer program. When the computer program is executed, it is specifically used to implement the model training method for semiconductor wafer defect detection disclosed in the first aspect of an embodiment of the present invention.

[0050] Based on the above-mentioned embodiments of the present invention, a model training method, system, electronic device and storage medium for semiconductor wafer defect detection are provided. The method is: based on a training data set, the probability of each defect category being sampled is determined, a training data stream is determined from the training data set according to the probability of the defect category being sampled, and the model is trained using the training data stream to obtain a defect detection model. The model training is performed by determining the probability of the obtained defect category being sampled and determining the training data stream from the training data set, thereby alleviating the problem of imbalanced training data, thereby improving the detection accuracy of the model for defect categories of semiconductor wafers. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0052] Figure 1 An example diagram of data of wafer defect images of various defect categories in an actual production process provided by an embodiment of the present invention;

[0053] Figure 2 A flowchart of a model training method for semiconductor wafer defect detection provided by an embodiment of the present invention;

[0054] Figure 3 A first flow chart of a defect detection model obtained by training provided by an embodiment of the present invention;

[0055] Figure 4 A second flow chart of a defect detection model obtained by training provided in an embodiment of the present invention;

[0056] Figure 5 A third flow chart of a defect detection model obtained by training provided by an embodiment of the present invention;

[0057] Figure 6 A structural block diagram of a model training system for semiconductor wafer defect detection provided by an embodiment of the present invention;

[0058] Figure 7 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0059] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0060] In this application, the terms "comprises", "comprising" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprising a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element.

[0061] The production process of semiconductor wafers is very complicated, and the defect categories of the semiconductor wafers produced are also many (usually there are hundreds of defect categories), and the characteristics of different defect categories are not very clear; due to the subjectivity and fatigue of humans, quality inspectors are less effective when manually re-evaluating defect categories. Therefore, deep learning-based methods are usually used to automatically identify defect categories.

[0062] However, the inventors have found through research that the defects generated in the actual production process of semiconductor wafers are often concentrated in a limited number of defect categories, such as Figure 1 As shown in the data example of wafer defect images of various defect categories in the actual production process, Figure 1 The horizontal axis in is the index of the defect category. Figure 1 The vertical axis in is the number of wafer defect images in the defect category. Figure 1 There are 5000 wafer defect images in defect category 1, and Figure 1 If there are 10 defect categories in the dataset, there are only 50 wafer defect images. If the deep learning model is trained according to the data generated in the actual production process of semiconductor wafers, the learning effects of the trained model for different defect categories will vary greatly. The larger the amount of data, the better the learning effect of the defect category, and vice versa. This will affect the model's detection results for the defect categories of semiconductor wafers.

[0063] To solve the above problems, this solution proposes a model training method, system, electronic device and storage medium for semiconductor wafer defect detection, which determines the probability of each defect category being sampled based on a training data set. According to the probability of the defect category being sampled, a training data stream is determined from the training data set, and the training data stream is used for model training to obtain a defect detection model. The training data stream is determined from the training data set based on the calculated probability of the defect category being sampled to perform model training, thereby alleviating the problem of unbalanced training data and improving the detection accuracy of the model for the defect category of semiconductor wafers.

[0064] See also Figure 2 , shows a flow chart of a model training method for semiconductor wafer defect detection provided by an embodiment of the present invention, the model training method comprising:

[0065] Step S201: determining the probability of each defect category being sampled based on the training data set.

[0066] It should be noted that a training data set is collected in advance, and the training data set contains wafer defect images of multiple defect categories.

[0067] After collecting the training data set, the wafer defect images of multiple defect categories in the training data set are pre-processed according to the particularities of the wafer defect images in the training data set; specifically, the pre-processing such as the zoom range, color change amplitude and translation amplitude of the wafer defect images in the training data set is set. The following is a detailed description of the pre-processing such as setting the zoom range, color change amplitude and translation amplitude.

[0068] It is understandable that the wafer defect images in the above-mentioned training data set have particularities such as "wafer defects are sensitive to defect size", "wafer defects are sensitive to color" and "wafer defects are sensitive to position".

[0069] Wafer defects are sensitive to defect size: that is, for the same defect form, if the defect size is greater than a certain threshold, it is judged as a wafer defect, and if the defect size is less than the threshold, it is judged as no wafer defect. When training models under the pytorch framework, the commonly used scaling-related function is RandomResizedCrop(), which randomly crops a smaller area from the wafer defect image and scales the area to the specified size.

[0070] The key parameter of the RandomResizedCrop() function is the scale parameter, which is used to control the ratio of the area of ​​the image after the function transformation to the area of ​​the original image. It can be set to ( ∈[α,β], β is generally taken as 0.1).

[0071] Assuming that the original image and the transformed image are both squares with side lengths of m and n respectively, the actual scaling range of the wafer defect image is , assuming that after testing, it is found that the safe edge length scaling range is [γ,δ], then , , substitute The scaling factor of the RandomResizedCrop() function can be obtained, and the scaling range of the wafer defect image is set by the scaling factor, thereby controlling the scaling of the wafer defect image within a controllable range.

[0072] Wafer defects are sensitive to color: color cast defects (such as oxidation defects) will show some color anomalies. When training the model, it is necessary to control the image color change amplitude (especially the change of hue). When training the model under the pytorch framework, the commonly used color-related function is ColorJitter (). ColorJitter () increases the generalization ability of the model by randomly adjusting the brightness, contrast, saturation and hue of the wafer defect image.

[0073] In the function ColorJitter ( ), the hue factor hue_factor is randomly sampled from [-hue, hue]. The maximum hue value within the controllable color range before and after the transformation can be determined experimentally. The maximum hue value is set to p, so the transformation range of hue_factor is [-p, p]. The color transformation amplitude of the wafer defect image is set in the above manner.

[0074] Wafer defects are sensitive to position: Wafer defects will appear first in the middle area of ​​the wafer defect image, so the amplitude of operations such as image translation needs to be controlled during model training. When training the model under the pytorch framework, the position-related function used is Translate (). Translate () is used to move all pixels in the wafer defect image in the horizontal and vertical directions according to the specified pixel value.

[0075] Assume that the model input size is a square with a side length of N. When using the Translate() function, the safe range of horizontal and vertical movement is [-N / 2, N / 2]. Considering that the defect position may be affected by other transformation operations during model training, a safety factor ε (ε∈[0,1.0]) is introduced on the basis of the range [-N / 2, N / 2]. Usually, ε∈[0,0.70] can be set. When the Translate() function is finally used, the range of horizontal and vertical movement distances can be [-ε*N / 2, ε*N / 2]. In addition, a spatial attention module can be introduced in the model structure design to improve the spatial perception ability of the model. The translation amplitude of the wafer defect image can be set in the above way.

[0076] In the specific implementation of step S201, the probability of each defect category being sampled is determined based on the training data set. When determining the probability of each defect category being sampled based on the training data set, this solution provides two probability determination methods, which are explained below respectively.

[0077] The first method to determine probability:

[0078] Based on the training data set, a first probability vector and a second probability vector are determined. The first probability vector and the second probability vector contain the probabilities of each defect category being sampled. The probabilities of at least two defect categories being sampled in the first probability vector are different, and the probabilities of each defect category being sampled in the second probability vector are the same.

[0079] Specifically, the specific method for determining the first probability vector and the second probability vector is as follows: based on the number of defect categories in the training data set (equivalent to the total number of categories), the number of wafer defect images in each defect category, and the first value of a preset parameter (denoted as q), determine the first probability vector; based on the number of defect categories in the training data set, the number of wafer defect images in each defect category, and the second value of the preset parameter, determine the second probability vector.

[0080] The “first value” mentioned above may be 1, and the “second value” may be 0.

[0081] It should be noted that when determining the probability of each defect category being sampled in the first probability vector and the second probability vector, the probability of the defect category being sampled can be determined by formula (1).

[0082] (1);

[0083] In formula (1), p jis the probability of defect category j being sampled in the training data set (equivalent to the probability of sampling from defect category j), n j is the number of wafer defect images in defect category j, C is the number of defect categories in the training data set, q is the preset parameter mentioned above, and q is a value between [0,1].

[0084] From formula (1), it can be seen that when q = 1 (the preset parameter is the first value), the probability of defect category j being sampled is At this time, it can be characterized that the probability of a certain defect category being sampled is proportional to the number of wafer defect images in the defect category, that is, the probability of a certain defect category being sampled in the first probability vector is related to the number of wafer defect images in the defect category. The greater the number of wafer defect images in defect category j, the greater the probability of defect category j being sampled (conversely, the smaller the probability of being sampled); the probabilities of at least two defect categories being sampled in the first probability vector are different.

[0085] When q=0 (the preset parameter is the second value), the probability of defect category j being sampled is , it can be represented that the probability of different defect categories being sampled is independent of the number of wafer defect images in the defect category, that is, the probability of each defect category being sampled in the second probability vector is the same.

[0086] It is worth noting that j∈[1,2,3,…,C] in the above formula (1), so before sampling from the training data set, the first probability vector and the second probability vector determined are both C-dimensional probability vectors; for example: the first probability vector determined is [0.01,0.3,0.23,…,0.04], and random sampling is performed from the training data set according to the size of the probability in the first probability vector to obtain the final sampled defect category.

[0087] Among them, the method of sampling from the training data set according to the first probability vector is called instance-balanced sampling, that is, the sampling method when q=1 can be called instance-balanced sampling; the method of sampling from the training data set according to the second probability vector is called class-balanced sampling, that is, the sampling method when q=0 can be called class-balanced sampling.

[0088] The above is an explanation of the first method of determining probability.

[0089] The second method to determine probability:

[0090] Based on the training data set and the root mean square calculation method, a third probability vector is determined, where the third probability vector includes the probability of each defect category being sampled.

[0091] Specifically, the specific method of determining the third probability vector is: using formula (2) to determine the probability of each defect category being sampled in the training data set to obtain a third probability vector, and the third probability vector contains the probability of each defect category being sampled determined by formula (2).

[0092] (2);

[0093] In formula (2), C is the number of defect categories in the training data set, p j is the probability of the jth defect category (equivalent to defect category j) being sampled in the training data set, n j is the number of wafer defect images in the jth defect category, n i is the number of wafer defect images in the i-th defect category.

[0094] It should be noted that j∈[1,2,3,…,C] in the above formula (2), so before sampling from the training data set, the determined third probability vector is a C-dimensional third probability vector; for example: the determined third probability vector is [0.01,0.3,0.23,…,0.04], and random sampling is performed from the training data set according to the probability in the third probability vector, so as to determine the defect category of the final sampling.

[0095] The above is an explanation of the second method of determining probability.

[0096] Step S202: determining a training data stream from the training data set according to the probability of the defect category being sampled, and performing model training using the training data stream to obtain a defect detection model, where the defect detection model is used to detect defect information of semiconductor wafers.

[0097] In the specific implementation of step S202 , after the probability of each defect category being sampled is calculated based on the training data set, a training data stream is determined from the training data set according to the calculated probability of each defect category being sampled.

[0098] After determining the training data stream from the training data set, data enhancement processing is performed on the training data stream based on the preset zoom range, color change amplitude, and translation amplitude; specifically, data enhancement processing (rotation, zoom, translation, color change, Mixup or Cutmix, etc.) is performed on the wafer defect images in the training data stream based on the preset zoom range, color change amplitude, and translation amplitude.

[0099] That is to say, when the wafer defect image is scaled, the color is changed and the translation is performed, the scaling, the color is changed and the translation is performed within the preset scaling range, the color change amplitude and the translation amplitude.

[0100] The model is trained using a determined training data stream, thereby training a defect detection model, which is used to detect defect information (defect category, defect location, etc.) of a semiconductor wafer.

[0101] In some embodiments, after the defect detection model is trained, the semiconductor wafer image to be inspected is input into the defect detection model for detection to obtain defect information (defect category, defect location, etc.) of the semiconductor wafer image to be inspected.

[0102] It can be seen from the content of the above step S201 that when determining the probability of each defect category being sampled based on the training data set, the first probability determination method can be used to calculate the probability of the defect category being sampled (calculating the first probability vector and the second probability vector), or the second probability determination method can be used to calculate the probability of the defect category being sampled (calculating the third probability vector);

[0103] When different probability determination methods are used, the methods of training the defect detection model are also different. Figures 3 to 5 Let’s explain in detail the different ways to train defect detection models.

[0104] In an embodiment of the present invention, the probability of a defect category being sampled is calculated to determine a training data stream from a training data set to perform model training, thereby alleviating the problem of imbalanced training data and improving the detection accuracy of the model for defect categories of semiconductor wafers.

[0105] For the above-mentioned embodiment of the present invention Figure 2 Step S202 involves training the defect detection model. When the first probability calculation method is used to calculate the probability of the defect category being sampled, refer to Figure 3 , shows a first flow chart of a defect detection model obtained by training provided by an embodiment of the present invention, Figure 3 The steps include:

[0106] Step S301: determining a first training data stream from a training data set according to a first probability vector.

[0107] In the process of specifically implementing step S301, a first training data stream is determined from the training data set according to the first probability vector, that is, the first training data stream is determined from the training data set in an "instance balanced sampling" manner, and the first training data stream includes a wafer defect image sampled from the training data set according to the first probability vector.

[0108] It should be noted that in the process of sampling from the training data set according to the first probability vector, at each sampling, the defect category of this sampling is first determined, and then a wafer defect image is randomly selected from the defect category of this sampling by balanced sampling (also called random sampling).

[0109] For example: in the process of sampling from the training data set according to the first probability vector, assuming that the defect category of this sampling is determined to be defect category 2, then a wafer defect image is randomly selected from defect category 2 in a balanced sampling manner; by performing multiple samplings in the above manner, the first training data stream can be obtained.

[0110] In some embodiments, after determining the first training data stream, data enhancement processing is performed on the first training data stream based on a preset zoom range, color change amplitude, and translation amplitude; specifically, data enhancement processing is performed on the wafer defect image in the first training data stream based on the preset zoom range, color change amplitude, and translation amplitude.

[0111] For example, data enhancement processing such as rotation, scaling, translation, color conversion, Mixup or Cutmix is ​​performed on the wafer defect image in the first training data stream.

[0112] It is worth noting that in the actual process of wafer defect detection, it is necessary to impose certain restrictions on the transformation amplitude of the wafer defect image transformation to keep the image label of the wafer defect image unchanged before and after the transformation; for example: if a customer site is sensitive to the size of the wafer defect (for the same defect, a large defect area is judged as a defect, and a small defect area is judged as no defect), then the amplitude of the wafer defect image scaling transformation must be controlled.

[0113] That is to say, when the wafer defect image is scaled, the color is changed and the translation is performed, the scaling, the color is changed and the translation is performed within the preset scaling range, the color change amplitude and the translation amplitude.

[0114] Step S302: According to the first training data stream and a preset number of training rounds, train the first model to obtain a second model, where the first model is a deep learning model.

[0115] In the specific implementation process of step S302, after determining the first training data stream, the first model (deep learning model) is trained according to the first training data stream and a preset number of training rounds to obtain the second model.

[0116] For example, the preset number of training rounds for training the first model is set to 90 rounds; according to the first training data stream, the preset model training method, and the preset number of training rounds (90 rounds), the first model is trained to obtain the second model.

[0117] Step S303: Determine a second training data stream from the training data set according to the second probability vector.

[0118] In the process of specifically implementing step S303, a second training data stream is determined from the training data set according to the second probability vector, that is, the second training data stream is determined from the training data set in a "class-balanced sampling" manner, and the second training data stream includes a wafer defect image sampled from the training data set according to the second probability vector.

[0119] It should be noted that in the process of sampling from the training data set according to the second probability vector, at each sampling, the defect category of this sampling is first determined, and then a wafer defect image is randomly selected from the defect category of this sampling by balanced sampling (also called random sampling).

[0120] For example: in the process of sampling from the training data set according to the second probability vector, assuming that the defect category of this sampling is determined to be defect category 4, then a wafer defect image is randomly selected from defect category 4 in a balanced sampling manner; by performing multiple samplings in the aforementioned manner, the second training data stream can be obtained.

[0121] In some embodiments, after determining the second training data stream, data enhancement processing is performed on the second training data stream based on a preset zoom range, color change amplitude, and translation amplitude; specifically, data enhancement processing (rotation, zoom, translation, color change, Mixup or Cutmix, etc.) is performed on the wafer defect image in the second training data stream based on the preset zoom range, color change amplitude, and translation amplitude.

[0122] Step S304: train the second model using the second training data stream to obtain a defect detection model.

[0123] In the specific implementation of step S304, after the second training stream is determined, the second model is trained by the second training data stream to obtain a defect detection model.

[0124] It can be understood that when the second model is trained using the second training data stream to obtain the defect detection model, the present solution provides at least three methods of training the second model using the second training data stream. These three methods are explained below respectively.

[0125] The first way to train the second model using the second training data stream:

[0126] After the second model is obtained through training, the backbone network layer of the second model is frozen, and only the last fully connected layer of the second model is retained for subsequent training, wherein the frozen backbone network layer has been trained with the first training data stream.

[0127] The training parameters of the fully connected layer of the second model with the backbone network layer frozen are randomly initialized, for example, the training parameters of the fully connected layer of the second model with the backbone network layer frozen are kaiming initialized.

[0128] Using the second training data stream, the second model with the backbone network layer frozen and the fully connected layer randomly initialized is trained until the second model converges (whether the model converges can be determined by setting training indicators), thereby obtaining a defect detection model.

[0129] That is, after freezing the backbone network layer of the second model and randomly initializing the fully connected layer of the second model, the second model is iteratively trained using the second training data stream (at this time only the last fully connected layer is retained for training), thereby obtaining a defect detection model.

[0130] It should be noted that in the first method of using the second training data stream to train the second model, it is necessary to perform steps such as determining the second training data stream, freezing the backbone network layer, randomly initializing the fully connected layer, and continuing to iteratively train the second model.

[0131] In practical applications, one execution order of the steps may be: freezing the backbone network layer of the second model, re-randomly initializing the training parameters of the fully connected layer of the second model, determining a second training data stream from the training data set according to the second probability vector, and continuing to iteratively train the second model using the second training data stream to obtain a defect detection model;

[0132] Another order of executing the steps may be: determining a second training data stream from the training data set according to the second probability vector, freezing the backbone network layer of the second model, randomly reinitializing the training parameters of the fully connected layer of the second model, and continuing to iteratively train the second model using the second training data stream to obtain a defect detection model.

[0133] The first method of using the second training data stream to train the second model is to use decoupling training technology to alleviate the problem of imbalanced training data during model training, or to use decoupling training technology to deal with the problem of data imbalance during semiconductor wafer defect detection, thereby improving the training accuracy of the model and improving the model's detection accuracy for semiconductor wafer defect categories.

[0134] The above is an explanation of the first method of training the second model using the second training data stream.

[0135] The second way to train the second model using the second training data stream:

[0136] The training parameters of the fully connected layer of the second model are scaled and the second model is trained based on the second training data stream to obtain a defect detection model.

[0137] It should be noted that the training parameters of the last fully connected layer of the second model are also called classifier parameters. The training parameters of the fully connected layer can be recorded as , where w i ∈R d is the training parameter (classifier parameter) of the fully connected layer corresponding to defect category i, w i ∈R d is a d-dimensional vector, and C is the number of defect categories in the training dataset.

[0138] After the second model is obtained through training, all training parameters of the second model are frozen.

[0139] Scale the training parameters of the fully connected layer of the second model with all training parameters frozen; specifically, multiply the training parameters of the fully connected layer of the second model with all training parameters frozen by a vector of the same dimension , thereby scaling the training parameters of the fully connected layer.

[0140] Furthermore, the training parameters of the fully connected layer of the second model with all training parameters frozen are scaled by formula (3).

[0141] (3);

[0142] In formula (3), is the training parameter (classifier parameter) of the fully connected layer corresponding to defect category i, For Vectors of the same dimension, i∈[1,2,3,…,C].

[0143] Based on the second training data stream, a second model with all training parameters frozen and training parameters of the fully connected layer parameters scaled is trained until the second model converges to obtain a defect detection model.

[0144] That is, after freezing all the training parameters of the second model and scaling the training parameters of the fully connected layer of the second model by formula (3), the second model with all the training parameters frozen and the training parameters of the fully connected layer scaled is iteratively trained using the second training data stream determined from the training data set by the “class balanced sampling” method until the second model converges to obtain a defect detection model.

[0145] For example: using the second training data stream determined from the training data set by the "class balanced sampling" method, the second model with all training parameters frozen and the training parameters of the fully connected layer scaled is iteratively trained for 10 rounds, and the second model converges to obtain a defect detection model.

[0146] It should be noted that, combined with the content of formula (3), it can be seen that in the process of iterative training of the second model with all training parameters frozen and the training parameters of the fully connected layer scaled, the training parameters at this time are only the vector , the training parameters (classifier parameters) of the final fully connected layer obtained by training are .

[0147] It should be further explained that in the second method of training the second model using the second training data stream, it is necessary to perform steps such as determining the second training data stream, freezing all training parameters of the second model, scaling the training parameters of the fully connected layer of the second model, and continuing to iterate the training of the second model.

[0148] In practical applications, one execution order of the steps may be: freezing all training parameters of the second model, scaling the training parameters of the fully connected layer of the second model, determining a second training data stream from the training data set according to the second probability vector, and continuing to iteratively train the second model using the second training data stream to obtain a defect detection model;

[0149] Another order of executing the steps may be: determining a second training data stream from the training data set according to the second probability vector, freezing all training parameters of the second model, scaling the training parameters of the fully connected layer of the second model, and continuing to iteratively train the second model using the second training data stream to obtain a defect detection model.

[0150] The second method of training the second model using the second training data stream, that is, using the classifier parameter rescaling technology to deal with the problem of data imbalance in the model training process, or using the classifier parameter rescaling technology to deal with the problem of data imbalance in the semiconductor wafer defect detection process, greatly alleviates the training difficulties caused by data imbalance, improves the training accuracy of the model, and improves the model's detection accuracy for semiconductor wafer defect categories.

[0151] The above is an explanation of the second method of training the second model using the second training data stream.

[0152] The third way to train the second model using the second training data stream:

[0153] The training parameters of the fully connected layer of the second model are normalized and the second model is trained based on the second training data stream to obtain a defect detection model.

[0154] It should be noted that the training parameters of the last fully connected layer of the second model are also called classifier parameters. The training parameters of the fully connected layer can be recorded as , where w i ∈R d is the training parameter (classifier parameter) of the fully connected layer corresponding to defect category i, wi ∈R d is a d-dimensional vector, and C is the number of defect categories in the training dataset.

[0155] After the second model is trained, the training parameters of the fully connected layer of the second model are normalized, that is, the training parameters (classifier parameters) of the fully connected layer of the second model obtained by training are normalized; specifically, w i ∈R d (Training parameters of the fully connected layer) Perform the normalization operation as shown in formula (4), and the normalized training parameters of the fully connected layer are recorded as .

[0156] (4);

[0157] In formula (4), τ is a model hyperparameter that controls the normalization strength. The value range of τ is (0, 1). The specific value of τ can be determined by experiments using the validation set. is L2 normalization.

[0158] Based on the second training data stream, a second model with normalized training parameters of the fully connected layer is trained until the second model converges to obtain a defect detection model.

[0159] That is, after the training parameters of the fully connected layer of the second model are normalized by formula (4), the second model with the normalized training parameters of the fully connected layer is iteratively trained using the second training data stream determined from the training data set by the “class balanced sampling” method until the second model converges to obtain a defect detection model.

[0160] For example, by using the second training data stream determined from the training data set by the “class balanced sampling” method, the second model with normalized training parameters of the fully connected layer is iteratively trained for 10 rounds until the second model converges to obtain a defect detection model.

[0161] It should be noted that in the third method of training the second model using the second training data stream, it is necessary to perform steps such as determining the second training data stream, normalizing the training parameters of the fully connected layer of the second model, and continuing to iterate the training of the second model.

[0162] In practical applications, one execution order of the steps may be: normalizing the training parameters of the fully connected layer of the second model, determining a second training data stream from the training data set according to the second probability vector, and continuing to iteratively train the second model using the second training data stream to obtain a defect detection model;

[0163] Another order of executing the steps may be: determining a second training data stream from the training data set according to the second probability vector, normalizing the training parameters of the fully connected layer of the second model, and continuing to iteratively train the second model using the second training data stream to obtain a defect detection model.

[0164] The third method of using the second training data stream to train the second model is to use the classifier parameter normalization technology to deal with the problem of data imbalance in the model training process, or to use the classifier parameter normalization technology to deal with the problem of data imbalance in the semiconductor wafer defect detection process. This greatly alleviates the training difficulties caused by data imbalance, improves the training accuracy of the model, and improves the model's detection accuracy for semiconductor wafer defect categories.

[0165] The above is an explanation of the third method of training the second model using the second training data stream.

[0166] Based on the contents of the above embodiments, the first application scenario of this solution is:

[0167] A1. According to the customer's requirements for wafer defect determination, a large number of wafer defect images are collected; the collected wafer defect images are counted by defect category to obtain the number of wafer defect images of each defect category, thereby obtaining a training data set.

[0168] A2. Design the first model according to actual needs and set initial parameters of the first model.

[0169] A3. Determine a first training data stream from the training data set in an “instance balanced sampling” manner, and train the first model according to the first training data stream, a preset model training method, and a preset number of training rounds (90 rounds) to obtain a second model.

[0170] A4. Freeze the backbone network layer of the second model, and perform kaiming initialization on the training parameters of the fully connected layer of the second model with the backbone network layer frozen; determine the second training data stream from the training data set in a "class-balanced sampling" manner; and continue to iteratively train the second model using the second training data stream (at this time, only retain the last fully connected layer for training) until the second model converges to obtain a defect detection model.

[0171] A5. After the defect detection model is obtained, the image to be tested is input into the defect detection model for inference to obtain defect information (such as defect type, defect location, etc.) of the image to be tested output by the defect detection model.

[0172] In the first application scenario of this solution, this solution introduces the decoupling training technology shown in the above-mentioned embodiments to effectively alleviate the data imbalance problem actually generated in the semiconductor wafer production process, improve the training accuracy of the deep learning model, and effectively improve the accuracy and speed of production line defect re-judgment when deploying and using the trained model; in addition, the decoupling training technology shown in the above-mentioned embodiments can be directly integrated into the model training code, and the iteration difficulty and maintenance difficulty of the model are relatively low.

[0173] Based on the contents of the above embodiments, the second application scenario of this solution is:

[0174] B1. According to the customer's requirements for wafer defect determination, a large number of wafer defect images are collected; the collected wafer defect images are counted by defect category to obtain the number of wafer defect images of each defect category, thereby obtaining a training data set.

[0175] B2. Design the first model according to actual needs and set initial parameters of the first model.

[0176] B3. Determine a first training data stream from the training data set in an “instance balanced sampling” manner, and train the first model according to the first training data stream, a preset model training method, and a preset number of training rounds (90 rounds) to obtain a second model.

[0177] B4. Freeze all training parameters of the second model, and scale the training parameters of the fully connected layer of the second model with all training parameters frozen; determine the second training data stream from the training data set in a "class balanced sampling" manner; use the second training data stream to iteratively train the second model with all training parameters frozen and the training parameters of the fully connected layer scaled until the second model converges to obtain a defect detection model.

[0178] B5. After obtaining the defect detection model, deploy the defect detection model to detect semiconductor wafer defects. During detection, input the image to be tested into the defect detection model for reasoning to obtain defect information of the image to be tested (such as defect category, defect location, etc.) output by the defect detection model.

[0179] In the second application scenario of the present solution, the present solution aims at the data imbalance problem actually generated in the semiconductor wafer production process, and introduces the classifier parameter rescaling technology shown in the above-mentioned embodiments to effectively alleviate the data imbalance problem, improve the training accuracy of the deep learning model, and effectively improve the accuracy and speed of production line defect re-judgment when deploying and using the trained model; in addition, the classifier parameter rescaling technology shown in the above-mentioned embodiments can be directly integrated into the model training code, and the iteration difficulty and maintenance difficulty of the model are relatively low.

[0180] Based on the contents of the above embodiments, the third application scenario of this solution is:

[0181] C1. According to the customer's requirements for wafer defect determination, a large number of wafer defect images are collected; the collected wafer defect images are counted by defect category to obtain the number of wafer defect images of each defect category, thereby obtaining a training data set.

[0182] C2. Design the first model according to actual needs and set initial parameters of the first model.

[0183] C3. Determine a first training data stream from the training data set in an “instance balanced sampling” manner, and train the first model according to the first training data stream, a preset model training method, and a preset number of training rounds (90 rounds) to obtain a second model.

[0184] C4. Normalize the training parameters of the fully connected layer of the second model; determine the second training data stream from the training data set in a "class balanced sampling" manner; use the second training data stream to iteratively train the second model with normalized training parameters of the fully connected layer until the second model converges to obtain a defect detection model.

[0185] C5. After the defect detection model is obtained, the defect detection model is deployed to detect semiconductor wafer defects. During detection, the image to be tested is input into the defect detection model for reasoning to obtain defect information of the image to be tested (such as defect category, defect location, etc.) output by the defect detection model.

[0186] In the third application scenario of this solution, this solution introduces the classifier parameter normalization technology shown in the above-mentioned embodiments to effectively alleviate the data imbalance problem actually generated in the semiconductor wafer production process, improve the training accuracy of the deep learning model, and effectively improve the accuracy and speed of production line defect re-judgment when deploying and using the trained model; in addition, the classifier parameter normalization technology shown in the above-mentioned embodiments can be directly integrated into the model training code, and the iteration difficulty and maintenance difficulty of the model are relatively low.

[0187] For the above-mentioned embodiment of the present invention Figure 2 Step S202 involves training the defect detection model. When the second probability determination method is used to determine the probability of the defect category being sampled, refer to Figure 4 , shows a second flow chart of a defect detection model obtained by training provided by an embodiment of the present invention, Figure 4 The steps include:

[0188] Step S401: determining a third training data stream from the training data set according to a third probability vector.

[0189] In the specific implementation of step S401, a third training data stream is determined from the training data set according to the third probability vector.

[0190] It should be noted that, in the process of sampling from the training data set according to the third probability vector, at each sampling, the defect category of this sampling is first determined, and then a wafer defect image is randomly selected from the defect category of this sampling.

[0191] For example: in the process of sampling from the training data set according to the third probability vector, assuming that the defect category of this sampling is determined to be defect category j, then a wafer defect image is randomly selected from defect category j; by performing multiple samplings in the above manner, the third training data stream can be obtained.

[0192] In some embodiments, after sampling the third training data stream, data enhancement processing is performed on the third training data stream based on a preset zoom range, color change amplitude, and translation amplitude; specifically, data enhancement processing is performed on the wafer defect image in the third training data stream based on a preset zoom range, color change amplitude, and translation amplitude.

[0193] For example, based on the preset zoom range, color change amplitude, and translation amplitude, the wafer defect image in the third training data stream is subjected to data enhancement processing such as rotation, zoom, translation, color change, Mixup, or Cutmix.

[0194] It is worth noting that in the actual process of wafer defect detection, it is necessary to impose certain restrictions on the transformation amplitude of the wafer defect image transformation to keep the image label of the wafer defect image unchanged before and after the transformation; for example: if a customer site is sensitive to the size of the wafer defect (for the same defect, a large defect area is judged as a defect, and a small defect area is judged as no defect), then the amplitude of the wafer defect image scaling transformation must be controlled.

[0195] Step S402: Use the third training data stream to train the deep learning model to obtain a defect detection model.

[0196] In the specific implementation process of step S402, the deep learning model is trained by the determined third training data stream until the deep learning model converges (whether the model converges is determined by specifying indicators), thereby obtaining a defect detection model.

[0197] The method of training to obtain the defect detection model given in step S401 and step S402 uses the root mean square sampling technology to deal with the problem of data imbalance in the model training process, or uses the root mean square sampling technology to deal with the problem of data imbalance in the semiconductor wafer defect detection process, which greatly alleviates the training difficulties caused by data imbalance and improves the training accuracy of the model.

[0198] Based on the contents of the above embodiments, the fourth application scenario of this solution is:

[0199] D1. According to the customer's requirements for wafer defect determination, a large number of wafer defect images are collected; the collected wafer defect images are counted by defect category to obtain the number of wafer defect images of each defect category, thereby obtaining a training data set.

[0200] D2. Design a deep learning model according to actual needs and set the initial parameters of the deep learning model.

[0201] D3. Determine the probability of each defect category being sampled according to the above formula (2) to obtain a third probability vector; perform data sampling once according to the third probability vector; and then perform data enhancement processing on the sampled wafer defect image.

[0202] D4. Repeat the data sampling and other processing in D3 to obtain a third training data stream; use the third training data stream to train the deep learning model until the deep learning model converges to obtain a defect detection model.

[0203] D5. Input the semiconductor wafer image to be inspected into the defect detection model for detection to obtain defect information of the semiconductor wafer image to be inspected.

[0204] This solution introduces root mean square sampling technology to effectively alleviate the data imbalance problem actually generated in the semiconductor wafer production process, improve the training accuracy of the deep learning model, and effectively improve the accuracy and speed of production line defect re-judgment when deploying and using the trained model; in addition, the root mean square sampling technology shown in the above embodiments can be directly integrated into the model training code, and the iteration difficulty and maintenance difficulty of the model are relatively low.

[0205] For the above-mentioned embodiment of the present invention Figure 2 Step S202 involves training the defect detection model. When the first probability determination method is used to calculate the probability of the defect category being sampled, refer to Figure 5 , shows a third flow chart of training a defect detection model provided by an embodiment of the present invention, Figure 5 The steps include:

[0206] Step S501: During the tth round of iterative training of the deep learning model, a fourth probability vector is determined based on the first probability vector, the second probability vector, the number of iterative training rounds t, and a preset total number of iterative rounds.

[0207] It should be noted that the iterative training round number t is greater than 0, and the initial value of the iterative training round number t is 1; the fourth probability vector includes the probability of each defect category being sampled.

[0208] In the process of specifically implementing step S501, during the tth round of iterative training of the deep learning model, based on the first probability vector, the second probability vector, the number of iterative training rounds t, and the total number of iterative rounds (denoted as T, the set value before training), the fourth probability vector used in the tth round of iterative training is calculated.

[0209] It should be noted that during the entire process of training the deep learning model, the deep learning model will undergo a total of T (total number of iterations) rounds of training, and the number of iteration training rounds t will increase as the iteration training progresses (the number of iteration training rounds t increases by 1 after each round of iteration training).

[0210] For example, when the deep learning model is subjected to the first round of iterative training, the number of iterative training rounds t is 1; after the first round of iterative training is completed, the deep learning model is subjected to the second round of iterative training, and the number of iterative training rounds t is 2; and so on, the number of iterative training rounds t increases as the iterative training progresses.

[0211] In some embodiments, the specific method for calculating the fourth probability vector used in the t-th round of iterative training is: during the t-th round of iterative training of the deep learning model, using formula (5), determine the probability of each defect category being sampled in the fourth probability vector used in the t-th round of iterative training.

[0212] (5);

[0213] In formula (5), t is the number of iterative training rounds, T is the preset total number of iterative rounds, is the probability of defect category j being sampled in the first probability vector, is the probability of defect category j being sampled in the second probability vector, is the probability of defect category j being sampled in the fourth probability vector calculated during the t-th round of iterative training.

[0214] It is worth noting that j∈[1,2,3,…,C] in the above formula (5), so before sampling from the training data set, the determined fourth probability vector is a C-dimensional probability vector; for example: the determined fourth probability vector is [0.01,0.3,0.23,…,0.04], and random sampling is performed from the training data set according to the size of the probability in the fourth probability vector to obtain the final sampled defect category.

[0215] Step S502: Determine a fourth training data stream from the training data set according to a fourth probability vector.

[0216] In the process of specifically implementing step S502, after calculating the fourth probability vector used in the t-th round of iterative training, data sampling is performed from the training data set according to the fourth probability vector, and the fourth training data stream used in the t-th round of iterative training can be determined. The fourth training data stream includes the wafer defect image sampled from the training data set according to the fourth probability vector.

[0217] It should be noted that in the process of sampling from the training data set according to the fourth probability vector, at each sampling, the defect category of this sampling is first determined, and then a wafer defect image is randomly selected from the defect category of this sampling by balanced sampling (also called random sampling).

[0218] For example: in the process of sampling from the training data set according to the fourth probability vector, assuming that the defect category of this sampling is determined to be defect category 2, then a wafer defect image is randomly selected from defect category 2 in a balanced sampling manner; by performing multiple samplings in the aforementioned manner, the fourth training data stream used in the tth round of iterative training can be obtained.

[0219] In some embodiments, after determining the fourth training data stream used in the tth round of iterative training, data enhancement processing is performed on the fourth training data stream based on a preset zoom range, color change amplitude, and translation amplitude; specifically, data enhancement processing is performed on the wafer defect image in the fourth training data stream based on the preset zoom range, color change amplitude, and translation amplitude.

[0220] For example, data enhancement processing such as rotation, scaling, translation, color conversion, Mixup or Cutmix is ​​performed on the wafer defect image in the fourth training data stream.

[0221] It is worth noting that in the actual process of wafer defect detection, it is necessary to impose certain restrictions on the transformation amplitude of the wafer defect image transformation to keep the image label of the wafer defect image unchanged before and after the transformation; for example: if a customer site is sensitive to the size of the wafer defect (for the same defect, a large defect area is judged as a defect, and a small defect area is judged as no defect), then the amplitude of the wafer defect image scaling transformation must be controlled.

[0222] Step S503: Iteratively train the deep learning model according to the fourth training data stream until the number of iterative training rounds t is greater than the total number of iterative rounds to obtain a defect detection model.

[0223] In the process of specifically implementing step S503, after sampling and obtaining the fourth training data stream used in the t-th round of iterative training, the deep learning model is subjected to the t-th round of iterative training according to the fourth training data stream; after completing the t-th round of iterative training, t=t+1 (i.e., the number of iterative training rounds t is increased by 1, which is equivalent to updating the number of iterative training rounds t).

[0224] Determine whether the updated iterative training round number t is greater than the total number of iterative rounds; if the updated iterative training round number t is not greater than the total number of iterative rounds, return to execute step S501; if the updated iterative training round number t is greater than the total number of iterative rounds, determine that the deep learning model has completed training and obtain a defect detection model.

[0225] In a specific implementation, after completing the tth round of iterative training and updating the number of iterative training rounds t, it is determined whether the updated number of iterative training rounds t is greater than the total number of iterative rounds; if the updated number of iterative training rounds t is not greater than the total number of iterative rounds (t≦T), it indicates that the deep learning model has not completed all rounds of training. At this time, the process returns to step S501 to continue determining a new fourth probability vector and continue the next round of training for the deep learning model.

[0226] If the updated iterative training round number t is greater than the total iterative round number (t>T), it indicates that the deep learning model has completed all rounds of training. At this time, it is determined that the deep learning model has completed training and a defect detection model is obtained (that is, the deep learning model after completing the last round of training).

[0227] That is, if the updated iterative training round number t is not greater than the total iterative round number (t≦T), the process returns to step S501 until the iterative training round number t is greater than the total iterative round number to obtain a defect detection model.

[0228] For example: Assume that the total number of iterations T is 90; after the 89th round (t=89 at this time) of iterative training of the deep learning model, t=t+1=89+1, and the updated t(90) is not greater than T. At this time, return to step S501 to continue to determine the fourth probability vector used in the 90th round of iterative training, and then determine the fourth training data stream used in the 90th round of iterative training from the training data set based on the fourth probability vector, and use the fourth training data stream to train the deep learning model for the 90th round. After the 90th round (t=90 at this time) of iterative training of the deep learning model, t=t+1=90+1; at this time, the updated t(91) is greater than T(90), and it is determined that the deep learning model that has completed the 90th round of iterative training is a defect detection model.

[0229] It can be seen from the above formula (5) and the contents of steps S501 to S503 that when the deep learning model just starts iterative training, since the value of the iterative training round number t is small, the “ "This item accounts for a small proportion. The probability of each defect category being sampled in the fourth probability vector is mainly determined by " " is determined by this item; as the training of the deep learning model continues, the number of iterative training rounds t will gradually increase, and the probability of each defect category being sampled in the fourth probability vector will slowly approach " "This item is tilted (i.e. " "The proportion of this item will gradually increase); the above method can better alleviate the training problem of data imbalance in the model training process.

[0230] The method of training to obtain a defect detection model given in step S501 and step S503 utilizes the progressive balanced sampling technology to deal with the problem of data imbalance in the model training process, or in other words, utilizes the progressive balanced sampling technology to deal with the problem of data imbalance in the semiconductor wafer defect detection process, which greatly alleviates the training difficulties caused by data imbalance and improves the training accuracy of the model.

[0231] Based on the contents of the above embodiments, the fifth application scenario of this solution is:

[0232] E1. According to the customer's requirements for wafer defect determination, a large number of wafer defect images are collected; the collected wafer defect images are counted by defect category to obtain the number of wafer defect images of each defect category, thereby obtaining a training data set.

[0233] E2. Design a deep learning model according to actual needs and set the initial parameters of the deep learning model.

[0234] E3. According to the above formula (1), determine the probability of each defect category being sampled when q=1 , thus obtaining the first probability vector.

[0235] E4. According to the above formula (1), determine the probability of each defect category being sampled when q=0 , thus obtaining the second probability vector.

[0236] E5, based on and According to the above formula (5), the probability of each defect category being sampled in the tth round of iterative training is determined , thus obtaining the fourth probability vector used in the t-th round of iterative training.

[0237] E6. According to the fourth probability vector determined in E5, determine the fourth training data stream used in the tth round of iterative training from the training data set, and perform data enhancement processing on the fourth training data stream; the initial value of the iterative training round number t is 1.

[0238] E7. Perform the t-th round of iterative training on the deep learning model using the fourth training data stream used in the t-th round of iterative training; update the number of iterative training rounds t (t=t+1), and repeat E5 and E6 until the deep learning model completes T rounds of iterative training (until t>T), and obtains a defect detection model.

[0239] E8. After obtaining the defect detection model, deploy the defect detection model to detect semiconductor wafer defects. During detection, input the semiconductor wafer image to be detected into the defect detection model for reasoning to obtain defect information (such as defect category, defect location, etc.) of the semiconductor wafer image to be detected output by the defect detection model.

[0240] This solution introduces a progressive balanced sampling technology to effectively alleviate the data imbalance problem actually generated in the semiconductor wafer production process, improve the training accuracy of the deep learning model, and effectively improve the accuracy and speed of production line defect re-judgment when deploying and using the trained model; in addition, the progressive balanced sampling technology shown in the above-mentioned embodiments can be directly integrated into the model training code, and the iteration difficulty and maintenance difficulty of the model are relatively low.

[0241] Corresponding to the model training method for semiconductor wafer defect detection provided by the above embodiment of the present invention, see Figure 6 , an embodiment of the present invention also provides a structural block diagram of a model training system for semiconductor wafer defect detection, the model training system includes: a determination unit 601 and a training unit 602;

[0242] The determination unit 601 is used to determine the probability of each defect category being sampled based on a training data set, where the training data set includes wafer defect images of multiple defect categories.

[0243] In some embodiments, the determination unit 601 is specifically used to: determine a first probability vector and a second probability vector based on a training data set, the first probability vector and the second probability vector contain the probabilities of each defect category being sampled, the probabilities of at least two defect categories being sampled in the first probability vector are different, and the probabilities of each defect category being sampled in the second probability vector are the same.

[0244] In some other embodiments, the determination unit 601 is specifically configured to: determine the probability of each defect category being sampled in the training data set using formula (2) to obtain a third probability vector.

[0245] In some specific embodiments, the process of determining the first probability vector and the second probability vector by the determination unit 601 includes: determining the first probability vector based on the number of defect categories in the training data set, the number of wafer defect images in each defect category, and a first value of a preset parameter; determining the second probability vector based on the number of defect categories in the training data set, the number of wafer defect images in each defect category, and a second value of a preset parameter.

[0246] The training unit 602 is used to determine a training data stream from the training data set according to the probability of the defect category being sampled, and use the training data stream to perform model training to obtain a defect detection model, which is used to detect defect information of the semiconductor wafer.

[0247] In some embodiments, the training unit 602 is specifically used to: determine a first training data stream from a training data set according to a first probability vector; train a first model to obtain a second model based on the first training data stream and a preset number of training rounds, wherein the first model is a deep learning model; determine a second training data stream from the training data set according to a second probability vector; and train the second model using the second training data stream to obtain a defect detection model.

[0248] In some other embodiments, the training unit 602 is specifically used to: determine a third training data stream from the training data set according to the third probability vector; and use the third training data stream to train the deep learning model to obtain a defect detection model.

[0249] In some other embodiments, the training unit 602 is specifically used to: during the tth round of iterative training of the deep learning model, determine a fourth probability vector based on the first probability vector, the second probability vector, the number of iterative training rounds t, and a preset total number of iterative rounds, the fourth probability vector including the probability of each defect category being sampled; determine a fourth training data stream from the training data set according to the fourth probability vector; and iteratively train the deep learning model according to the fourth training data stream until the number of iterative training rounds t is greater than the total number of iterative rounds to obtain a defect detection model.

[0250] Specifically, during the t-th round of iterative training of the deep learning model, the fourth probability vector is determined using formula (5).

[0251] In some specific embodiments, the process in which the training unit 602 uses the second training data stream to train the second model to obtain a defect detection model includes: freezing the backbone network layer of the second model; randomly initializing the training parameters of the fully connected layer of the second model with the backbone network layer frozen; and using the second training data stream to train the second model with the backbone network layer frozen and the fully connected layer randomly initialized until the second model converges to obtain a defect detection model.

[0252] In other specific embodiments, the process in which the training unit 602 uses the second training data stream to train the second model to obtain a defect detection model includes: freezing all training parameters of the second model; scaling the training parameters of the fully connected layer of the second model with all training parameters frozen; and based on the second training data stream, training the second model with all training parameters frozen and the training parameters of the fully connected layer scaled until the second model converges to obtain a defect detection model.

[0253] In some other specific embodiments, the process in which the training unit 602 uses the second training data stream to train the second model to obtain a defect detection model includes: normalizing the training parameters of the fully connected layer of the second model; based on the second training data stream, training the second model with the normalized training parameters of the fully connected layer until the second model converges to obtain the defect detection model.

[0254] Preferably, combined Figure 6 The model training system further includes:

[0255] A setting unit is used to set the zoom range, color change amplitude and translation amplitude of the wafer defect image in the training data set.

[0256] In an embodiment of the present invention, the probability of a defect category being sampled is calculated and a training data stream is sampled from a training data set to perform model training, thereby alleviating the problem of imbalanced training data and improving the detection accuracy of the model for defect categories of semiconductor wafers.

[0257] An embodiment of the present invention also provides a computer storage medium for storing a computer program. When the computer program is executed, it is specifically used to implement the model training method for semiconductor wafer defect detection provided by the above method embodiment.

[0258] The present invention also provides an electronic device, see Figure 7 , Figure 7 is a schematic diagram of the structure of the electronic device, which may include a memory 701 and a processor 702;

[0259] The memory 701 is used to store computer programs;

[0260] The processor 702 is used to execute a computer program, specifically to implement the model training method for semiconductor wafer defect detection provided by the above method embodiment.

[0261] In summary, the embodiments of the present invention provide a model training method, system, electronic device and storage medium for semiconductor wafer defect detection. Model training is performed by sampling a training data stream from a training data set according to the calculated probability of the defect category being sampled, thereby alleviating the problem of imbalanced training data and improving the model's detection accuracy for semiconductor wafer defect categories.

[0262] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can refer to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiment. The system and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without creative work.

[0263] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0264] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A model training method for semiconductor wafer defect detection, characterized in that: The method comprises: Determine the probability of each defect category being sampled based on a training data set, wherein the training data set includes a plurality of wafer defect images of the defect category; A training data stream is determined from the training data set according to the probability of the defect category being sampled, and the training data stream is used to perform model training to obtain a defect detection model, where the defect detection model is used to detect defect information of a semiconductor wafer.

2. The method according to claim 1, characterized in that Determine the probability of each defect category being sampled based on the training data set, including: Based on the training data set, a first probability vector and a second probability vector are determined, wherein the first probability vector and the second probability vector contain the probabilities of each of the defect categories being sampled, the probabilities of at least two of the defect categories being sampled in the first probability vector are different, and the probabilities of each of the defect categories being sampled in the second probability vector are the same.

3. The method according to claim 1, characterized in that Determine the probability of each defect category being sampled based on the training data set, including: use Determine the probability of each defect category being sampled in the training data set to obtain a third probability vector, wherein the third probability vector contains the probability of each defect category being sampled, C is the number of defect categories in the training data set, and p j is the probability that the j-th defect category in the training data set is sampled, n j is the number of wafer defect images in the jth defect category, n i is the number of wafer defect images in the i-th defect category.

4. The method according to claim 2, characterized in that: Determining a training data stream from the training data set according to the probability of the defect category being sampled, and performing model training using the training data stream to obtain a defect detection model, including: Determining a first training data stream from the training data set according to the first probability vector; Training a first model to obtain a second model according to the first training data stream and a preset number of training rounds, wherein the first model is a deep learning model; determining a second training data stream from the training data set according to the second probability vector; The second model is trained using the second training data stream to obtain a defect detection model.

5. The method according to claim 3, characterized in that: Determining a training data stream from the training data set according to the probability of the defect category being sampled, and performing model training using the training data stream to obtain a defect detection model, including: Determining a third training data stream from the training data set according to the third probability vector; The deep learning model is trained using the third training data stream to obtain a defect detection model.

6. The method according to claim 2, characterized in that Determining a training data stream from the training data set according to the probability of the defect category being sampled, and performing model training using the training data stream to obtain a defect detection model, including: In the process of performing the tth round of iterative training on the deep learning model, based on the first probability vector, the second probability vector, the number of iterative training rounds t, and the preset total number of iterative rounds, a fourth probability vector is determined, wherein the fourth probability vector includes the probability of each of the defect categories being sampled; Determining a fourth training data stream from the training data set according to the fourth probability vector; The deep learning model is iteratively trained according to the fourth training data stream until the number of iterative training rounds t is greater than the total number of iterative rounds to obtain a defect detection model.

7. The method according to claim 2, characterized in that: Based on the training data set, determining a first probability vector and a second probability vector includes: Determine a first probability vector based on the number of defect categories in the training data set, the number of wafer defect images in each defect category, and a first value of a preset parameter; A second probability vector is determined based on the number of defect categories in the training data set, the number of wafer defect images in each defect category, and the second value of the preset parameter.

8. The method according to claim 4, characterized in that Using the second training data stream to train the second model to obtain a defect detection model includes: Freeze the backbone network layer of the second model; Randomly initializing the training parameters of the fully connected layer of the second model with the backbone network layer frozen; The second model with the backbone network layer frozen and the fully connected layer randomly initialized is trained using the second training data stream until the second model converges to obtain a defect detection model.

9. The method according to claim 4, characterized in that Using the second training data stream to train the second model to obtain a defect detection model includes: Freeze all training parameters of the second model; Scaling the training parameters of the fully connected layer of the second model with all training parameters frozen; Based on the second training data stream, the second model with all training parameters frozen and training parameters of the fully connected layer scaled is trained until the second model converges to obtain a defect detection model.

10. The method according to claim 4, characterized in that Using the second training data stream to train the second model to obtain a defect detection model includes: Normalizing the training parameters of the fully connected layer of the second model; Based on the second training data stream, the second model with normalized training parameters of the fully connected layer is trained until the second model converges to obtain a defect detection model.

11. The method according to claim 6, characterized in that In the process of performing the tth round of iterative training on the deep learning model, determining a fourth probability vector based on the first probability vector, the second probability vector, the number of iterative training rounds t, and a preset total number of iterative rounds, includes: During the tth round of iterative training of the deep learning model, , determine the fourth probability vector, where T is the preset total number of iterations, is the probability of defect category j being sampled in the first probability vector, is the probability of defect category j being sampled in the second probability vector, is the probability of defect category j being sampled in the fourth probability vector calculated during the t-th round of iterative training.

12. The method according to any one of claims 1 to 11, characterized in that: Before determining the probability of each defect category being sampled based on the training data set, it also includes: Set the zoom range, color transformation amplitude, and translation amplitude of the wafer defect images in the training dataset.

13. A model training system for semiconductor wafer defect detection, characterized in that: The system comprises: A determination unit, configured to determine the probability of each defect category being sampled based on a training data set, wherein the training data set includes a plurality of wafer defect images of the defect category; A training unit is used to determine a training data stream from the training data set according to the probability of the defect category being sampled, and use the training data stream to perform model training to obtain a defect detection model, wherein the defect detection model is used to detect defect information of a semiconductor wafer.

14. An electronic device, characterized in that: including memory and processor; The memory is used to store computer programs; The processor is used to execute the computer program, specifically to implement the model training method for semiconductor wafer defect detection as described in any one of claims 1 to 12.

15. A computer storage medium, characterized in that Used to store a computer program, which, when executed, is specifically used to implement the model training method for semiconductor wafer defect detection as described in any one of claims 1 to 12.

Images