Detection model determination method, device and system and detection method
By reweighting the loss value of the deep learning model in semiconductor wafer defect detection, the poor training effect caused by data imbalance is solved, the detection accuracy is improved, and better model accuracy is achieved.
Patent Information
- Application Number
- CN202310747946.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-06-21
- Publication Date
- 2024-12-31
AI Technical Summary
In the detection of semiconductor wafer defects, existing deep learning models have poor training results due to data imbalance, especially defect categories with small data volumes, which leads to poor detection accuracy.
By reweighting the loss values of multiple sample images, the loss scaling coefficient is used to enable different categories of samples to reach an equalization state during the training process, thereby optimizing the detection model and forming an equalized training data set to improve the model accuracy.
Through loss value reweighting processing, the balance of different categories of samples at the training level is achieved, the training accuracy and detection accuracy of deep learning models are improved, and the problem of poor model training effect caused by data imbalance is solved.
Smart Images
Figure CN119228709A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of semiconductor manufacturing technology, and in particular to a detection model determination method, a detection model determination device, a detection model determination system and a detection method. Background Art
[0002] Since the production process of semiconductor products such as semiconductor wafers is complex, many types of defects are generated (generally up to hundreds of types), and the characteristics of different defects are often not very clear. Affected by the clarity of wafer features and the subjectivity and fatigue of manual inspection, the effect of manual review is often not very good. Therefore, it is necessary to adopt a deep learning-based method to automatically identify defects. However, in actual production, defects are often concentrated in a limited number of categories, and the amount of data in different categories varies greatly. If traditional model training methods are used to train deep learning models, the learning effect of the model on different categories of defects will also vary greatly. The defect category with a larger data volume has a better learning effect, and the defect category with a smaller data volume has a worse learning effect, resulting in poor detection performance and poor accuracy of the final trained model.
[0003] Therefore, there is an urgent need to provide a new detection model determination solution to solve the problem of poor model training effect caused by the above-mentioned data imbalance. Summary of the invention
[0004] Therefore, in order to overcome at least some of the defects and problems in the prior art, the embodiments of the present invention provide a detection model determination method, a detection model determination device, a detection model determination system and a detection model obtained by a detection method, which can effectively improve the training accuracy of the deep learning model while improving the detection accuracy.
[0005] Specifically, on the one hand, an embodiment of the present invention provides a detection model determination method, including: obtaining a sample image set, wherein the sample image set includes multiple sample images; using the sample image set to iteratively process the detection model; wherein each iteration of the iterative processing includes: using the detection model to obtain multiple loss values corresponding one-to-one to the multiple sample images; re-weighting the multiple loss values to obtain multiple calculated loss values corresponding one-to-one to the multiple sample images; and optimizing the detection model or completing the iterative processing according to the multiple calculated loss values.
[0006] In one embodiment, the detection model determination method also includes: obtaining a plurality of loss scaling coefficients corresponding to the sample image set based on the plurality of sample images; the re-weighting processing of the plurality of loss values to obtain a plurality of calculated loss values corresponding one-to-one to the plurality of sample images includes: performing the re-weighting processing on the plurality of loss values according to the plurality of loss scaling coefficients to obtain a plurality of calculated loss values corresponding one-to-one to the plurality of sample images.
[0007] In one embodiment, obtaining multiple loss scaling factors corresponding to the sample image set based on the multiple sample images includes: determining the multiple loss scaling factors based on multiple sample types corresponding to the multiple sample images; the multiple loss scaling factors correspond one-to-one to the multiple sample types.
[0008] In one embodiment, the multiple sample images include a target sample image, the multiple sample types include a target type corresponding to the target sample image; the multiple sample images include a target number of sample images corresponding to the target type; and the multiple loss scaling factors are determined according to the multiple sample types of the multiple sample images, including: determining the target loss scaling factor according to the target number
[0009] In one embodiment, the target loss scaling factor is a ratio of a preset base number to the target number.
[0010] In one embodiment, any two loss scaling factors among the multiple loss scaling factors satisfy: the multiple sample types include a first type corresponding to the first scaling factor and a second type corresponding to the second scaling factor; the multiple sample images include a first number of sample images corresponding to the first type and a second number of sample images corresponding to the second type; the first number is smaller than the second number, and the first scaling factor is larger than the second scaling factor.
[0011] In one embodiment, the multiple sample images include a target sample image, the multiple loss scaling factors include a target scaling factor corresponding to the target sample image, and the multiple loss values include a target loss value corresponding to the target sample image; the multiple loss values are subjected to the reweighting processing according to the multiple loss scaling factors to obtain a plurality of calculated loss values corresponding one-to-one to the multiple sample images, including: obtaining a target calculated loss value corresponding to the target loss value according to the target loss value and the target scaling factor.
[0012] In one embodiment, the target calculated loss value is the product of the target loss value and the target loss scaling factor.
[0013] An embodiment of the present invention provides a detection model determination device, comprising: a sample image acquisition module, used to acquire a sample image set, the sample image set including multiple sample images; an iterative processing module, used to iteratively process the detection model using the sample image set; wherein each iteration of the iterative processing comprises: using the detection model to acquire multiple loss values corresponding one-to-one to the multiple sample images; performing re-weighting processing on the multiple loss values to obtain multiple calculated loss values corresponding one-to-one to the multiple sample images; and optimizing the detection model or completing the iterative processing according to the multiple calculated loss values.
[0014] An embodiment of the present invention provides a workpiece detection method, comprising: acquiring a test image of the workpiece to be detected; and detecting the test image using a detection model, wherein the detection model is trained by any of the aforementioned model training methods.
[0015] An embodiment of the present invention provides a detection model determination system, including a processor and a memory connected to the processor, the memory storing instructions executed by the processor, and the instructions causing the processor to perform operations to perform a detection model determination method as described in any one of the above.
[0016] An embodiment of the present invention provides a readable medium, wherein the readable medium stores computer-readable instructions, wherein the computer-readable instructions include instructions for executing the detection model determination method described in any of the aforementioned embodiments.
[0017] It can be seen from the above that the above embodiments of the present invention can achieve one or more of the following beneficial effects: by re-weighting the multiple loss values corresponding to the multiple sample images during the training process, the samples of different categories are finally balanced at the training loss level, and the detection model is optimized based on the loss value, so that a balanced training data set can be indirectly obtained, and the deep learning model based on this balanced training data set can achieve better accuracy. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0019] Figure 1 This is a schematic diagram of the number of some defects in actual production.
[0020] Figure 2 The figure is a flow chart of a method for determining a detection model in one embodiment of the present invention.
[0021] Figure 3 FIG. 4 is a schematic diagram of various sample types of a sample image set in an embodiment of the present invention.
[0022] Figure 4 A schematic diagram of the structure of a detection model determination device provided by an embodiment of the present invention.
[0023] Figure 5 A schematic diagram of the structure of a detection model determination system provided by an embodiment of the present invention.
[0024] Figure 6 A schematic diagram of the structure of a readable medium provided for one embodiment of the present invention. DETAILED DESCRIPTION
[0025] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.
[0026] In order to enable those skilled in the art to better understand the technical solution of the present invention, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work should fall within the scope of protection of the present invention.
[0027] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0028] It should also be noted that the division of multiple embodiments in the present invention is only for the convenience of description and should not constitute a special limitation. The features in various embodiments can be combined and referenced to each other without contradiction.
[0029] [First embodiment]
[0030] like Figure 1The figure shows the number of defects in actual production. The horizontal axis is the defect category index, and the vertical axis is the amount of data corresponding to the defect. It can be seen that the largest category 1 has 5,000 items, while the smallest category 10 has only about 50 items. According to the traditional method, the deep learning model is directly trained on this data. The learning effect of the model on different categories of defects will vary greatly. The learning effect is better for defect categories with larger data volumes, such as category 1, and the learning effect is worse for defect categories with smaller data volumes, such as category 10, which will eventually lead to poor detection results. For this reason, refer to Figure 2 The first embodiment of the present invention provides a detection model determination method, comprising:
[0031] Step S1: obtaining a sample image set, wherein the sample image set includes a plurality of sample images;
[0032] Step S3: using the sample image set to iteratively process the detection model;
[0033] Among them, each iteration of the iterative processing in step S3 includes: S(3,1): using the detection model to obtain multiple loss values corresponding to the multiple sample images one by one; S(3,2): re-weighting the multiple loss values to obtain multiple calculated loss values corresponding to the multiple sample images one by one; S(3,3): optimizing the detection model according to the multiple calculated loss values or completing the iterative processing.
[0034] The detection model obtained by training through the detection model determination method can be used, for example, for feature detection of workpieces such as wafers, for example, for detecting defect types of workpieces.
[0035] The sample image set (also referred to as a training data set) obtained in step S1 includes multiple sample images, for example, multiple workpiece images, for example, multiple wafer defect images. The multiple sample images correspond to multiple sample types, and the multiple sample types are, for example, different defect types, such as Figure 3 As shown in , the defects can be divided into scratches, residual glue, over-exposure, damaged glue surface, etc. according to the type of defects. Alternatively, they can be classified according to the location, size, shape, etc. of the defects. This embodiment is not limited to this. The choice of defect type can be based on the type of classification that needs to be determined according to customer requirements. The loss value corresponding to each sample image in step S (3, 1) is the loss value calculated by the loss function after each sample image enters the detection model. The loss value can represent the difference between the detection result of the corresponding sample image through the detection model and the actual result (or expected result) of the sample image. In the related technology, the detection model can be optimized according to the loss value in the iterative processing of the detection model. For the specific explanation of the loss value, please refer to the relevant concepts in traditional deep learning, which will not be explained in detail in this embodiment. In the traditional deep learning method, the model is optimized directly by the loss value. For similar Figure 1 In the case of unbalanced data volume as shown, there are different learning effects for different data volumes, resulting in the problem of poor final detection accuracy. Therefore, in this embodiment, multiple loss values are re-weighted through step S(3,2) to obtain multiple calculated loss values to achieve a balanced state at the training loss level, so that a balanced training data set can be indirectly matched. The deep learning model based on this balanced training data set can achieve better accuracy. In addition, in this embodiment, there is no need to perform explicit processing (such as offline enhancement processing) on the sample image set, which makes the implementation simpler and more effective. Offline enhancement processing, for example, includes cropping, rotating, and other post-processing increments on samples with smaller data volumes. This method will undoubtedly increase the amount of data processing, so Figure 1 For example, if the data volume of category 10 is increased from 50 to 5000, which is consistent with category 1, the overall data volume will increase significantly. However, in this embodiment, such an increase in data volume will not occur, and the operation difficulty is lower. In step S(3,3), it can be determined whether the iterative processing is completed based on multiple calculated loss values, for example, multiple calculated loss values are compared with a threshold, etc. If it is greater than the threshold, the iteration continues and the detection model is optimized. If it is less than the threshold, the iterative processing is completed. The method for optimizing the detection model can be based on an existing optimization method.
[0036] Of course, step S3 also includes, for example, determining the detection model to be optimized according to requirements, and initializing parameters related to iterative processing of the detection model.
[0037] In some embodiments, before step S3, it also includes: step S2: obtaining multiple loss scaling coefficients corresponding to the sample image set according to the multiple sample images. Step S(3,2) specifically includes S(3,21) performing the re-weighting processing on the multiple loss values according to the multiple loss scaling coefficients to obtain multiple calculated loss values corresponding one-to-one to the multiple sample images. Step S(3,2) more specifically includes S(3,211): determining the multiple loss scaling coefficients according to the multiple sample types corresponding to the multiple sample images; the multiple loss scaling coefficients correspond one-to-one to the multiple sample types. Before step S(3,2), for example, it also includes labeling the multiple sample images so that each sample image has a corresponding type label. Or sample images of different sample types are placed in different folders.
[0038] For example Figure 3 As shown, the sample images of scratch defects can be placed in the scratch folder, and the sample images of residual adhesive defects can be placed in the residual adhesive folder. For example, multiple sample images correspond to k sample types, and different sample types can be recorded as the first category, the second category, the third category...the k-1 category, the kth category. The number of sample images belonging to the i-th category (i is 1 to k) among the multiple sample images (i.e., the number of i-th categories) is recorded as ni , the number of sample images of each sample type is recorded as n1, n2, n3, ... n k-1 、n k .
[0039] The plurality of sample images include a target sample image, the plurality of sample types include a target type corresponding to the target sample image, and the plurality of sample images include sample images of target numbers corresponding to target types. Step S (3, 211) specifically includes: determining the target loss scaling factor according to the target number.
[0040] Each sample image in the plurality of sample images can be used as a target sample image to obtain a loss scaling factor corresponding to each sample image. For example, the target type corresponding to the target sample image is the tth class, and the number of sample images of the tth class is n. t , that is, the target number is n t In step S(3,211), n t Determine the target loss scaling factor for the target sample image.
[0041] In one embodiment, any two loss scaling factors among the multiple loss scaling factors satisfy: the multiple sample types include a first type corresponding to the first scaling factor and a second type corresponding to the second scaling factor; the multiple sample images include a first number of sample images corresponding to the first type and a second number of sample images corresponding to the second type; the first number is less than the second number, and the first scaling factor is greater than the second scaling factor. For example, the first type corresponding to the first scaling factor is the pth type, and the second type corresponding to the second scaling factor is the qth type, where p and q both belong to 1 to k. If n p <n q , then the first scaling factor is greater than the second scaling factor. In other words, the sample type with a larger number of sample images has a larger loss scaling factor, and the sample type with a smaller number of sample images has a smaller loss scaling factor.
[0042] More specifically, for example, the target loss scaling factor is the ratio of the preset base to the target number. For example, the preset base is T, and the target number is n t , then the target loss scaling factor is T / n t The loss scaling factors corresponding to multiple sample types are T / n1, T / n2, T / n3, ... T / n k-1 , T / n k The preset cardinality T can be based on n1~n k Determine, for example, it can be n1~n k The maximum value of n1~n kFurthermore, the target calculation loss value is the product of the target loss value and the target loss scaling factor.
[0043] This embodiment also provides a detection method, which can be used for feature detection of workpieces such as wafers, specifically, for example, wafer defect detection. The detection method includes: acquiring a test image of the workpiece to be tested; using a detection model to detect the test image, and the detection model is obtained by any of the detection model determination methods described above.
[0044] Using the detection model obtained by the detection model determination method provided in the above embodiment of the present invention for detection can solve the problem of the imbalance in the number of sample images of different types, so that the learning effect of the detection model is better and the detection is more accurate.
[0045] [Second embodiment]
[0046] Reference Figure 4 The second embodiment of the present invention provides a detection model determination device 100, including a sample image acquisition module 11 and an iterative processing module 13. The training data module 11 is used to execute step S1 in the aforementioned detection model determination method, and the iterative processing module 13 is used to execute step S3 in the aforementioned detection model determination method. For the specific principles of the detection model determination device 100 provided in this embodiment, reference can be made to the description in the aforementioned first embodiment. The detection model determination device 100 provided in this embodiment is used to execute the aforementioned detection model determination method, and has the same beneficial effects as the aforementioned first embodiment, which will not be repeated here.
[0047] [Third embodiment]
[0048] Reference Figure 5 The third embodiment of the present invention provides a detection model determination system 200, which includes, for example, a processor 201 and a memory 202 electrically connected to the processor 201, wherein the memory 202 stores instructions executed by the processor 201, and the instructions enable the processor 201 to perform operations to perform any one of the detection model determination methods in the aforementioned first embodiment.
[0049] The functions and technical effects of the detection model determination system 200 provided in the third embodiment of the present invention can be referred to the relevant description of the detection model determination method in the aforementioned first embodiment, and will not be repeated here.
[0050] [Fourth embodiment]
[0051] Reference Figure 6 The fourth embodiment of the present invention provides a readable medium 300, which stores computer-readable instructions 301. The computer-readable instructions 301 include instructions for executing any one of the detection model determination methods in the first embodiment.
[0052] The functions and technical effects of the readable medium 300 provided in the fourth embodiment of the present invention can refer to the technical effects of the detection model determination method in the aforementioned first embodiment, and will not be repeated here.
[0053] It should be noted that in the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and / or methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units / modules is only a logical function division. There may be other division methods in actual implementation, such as multiple units or modules can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.
[0054] The units / modules described as separate components may or may not be physically separated, and the components shown as units / modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units / modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0055] In addition, each functional unit / module in each embodiment of the present invention may be integrated into one processing unit / module, or each unit / module may exist physically separately, or two or more units / modules may be integrated into one unit / module. The above-mentioned integrated unit / module may be implemented in the form of hardware or in the form of hardware plus software functional units / modules.
[0056] The above-mentioned integrated unit / module implemented in the form of a software functional unit / module can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for causing one or more processors of a computer device (which can be a personal computer, a server, or a network device, etc.) to perform some steps of the method described in each embodiment of the present invention.
[0057] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any form. Although the present invention has been disclosed as a preferred embodiment as above, it is not used to limit the present invention. Any technician familiar with this profession can make some changes or modifications to equivalent embodiments of equivalent changes using the technical contents disclosed above without departing from the scope of the technical solution of the present invention. However, any simple modification, equivalent change and modification made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention still fall within the scope of the technical solution of the present invention.
Claims
1. A method for determining a detection model, characterized in that: include: Acquire a sample image set, wherein the sample image set includes a plurality of sample images; The detection model is iteratively processed using the sample image set; wherein each iteration of the iterative processing includes: using the detection model to obtain multiple loss values corresponding one-to-one to the multiple sample images; re-weighting the multiple loss values to obtain multiple calculated loss values corresponding one-to-one to the multiple sample images; optimizing the detection model according to the multiple calculated loss values or completing the iterative processing.
2. The detection model determination method according to claim 1, characterized in that: Also includes: Acquire a plurality of loss scaling coefficients corresponding to the sample image set according to the plurality of sample images; The re-weighting processing of the multiple loss values to obtain multiple calculated loss values corresponding one-to-one to the multiple sample images includes: performing the re-weighting processing on the multiple loss values according to the multiple loss scaling coefficients to obtain multiple calculated loss values corresponding one-to-one to the multiple sample images.
3. The detection model determination method according to claim 2, characterized in that: The acquiring of multiple loss scaling factors corresponding to the sample image set according to the multiple sample images includes: determining the multiple loss scaling factors according to multiple sample types corresponding to the multiple sample images; and the multiple scaling factors correspond one-to-one to the multiple sample types.
4. The detection model determination method according to claim 3, characterized in that: The multiple sample images include a target sample image, the multiple sample types include a target type corresponding to the target sample image; the multiple sample images include a target number of sample images corresponding to the target type; The determining the multiple loss scaling factors according to the multiple sample types of the multiple sample images includes: determining the target loss scaling factor according to the target number.
5. The detection model determination method according to claim 4, characterized in that: The target loss scaling factor is a ratio of a preset base number to the target number.
6. The detection model determination method according to claim 3, characterized in that: Any two loss scaling factors among the multiple loss scaling factors satisfy: the multiple sample types include a first type corresponding to the first scaling factor and a second type corresponding to the second scaling factor; the multiple sample images include a first number of sample images corresponding to the first type and a second number of sample images corresponding to the second type; the first number is smaller than the second number, and the first scaling factor is larger than the second scaling factor.
7. The detection model determination method according to claim 2, characterized in that: The multiple sample images include a target sample image, the multiple loss scaling factors include a target scaling factor corresponding to the target sample image, and the multiple loss values include a target loss value corresponding to the target sample image; The multiple loss values are subjected to the re-weighting processing according to the multiple loss scaling coefficients to obtain multiple calculated loss values corresponding one-to-one to the multiple sample images, including: obtaining a target calculated loss value corresponding to the target loss value according to the target loss value and the target scaling coefficient.
8. The detection model determination method according to claim 7, characterized in that: The target calculated loss value is the product of the target loss value and the target loss scaling factor.
9. A detection model determination device, characterized in that: include: A sample image acquisition module, used to acquire a sample image set, wherein the sample image set includes a plurality of sample images; An iterative processing module is used to iteratively process the detection model using the sample image set; wherein each iteration of the iterative processing includes: using the detection model to obtain multiple loss values corresponding to the multiple sample images one by one; re-weighting the multiple loss values to obtain multiple calculated loss values corresponding to the multiple sample images one by one; optimizing the detection model according to the multiple calculated loss values or completing the iterative processing.
10. A workpiece detection method, characterized in that: include: Acquire the image of the workpiece to be tested; The image to be tested is tested using a detection model, wherein the detection model is obtained by a detection model determination method according to any one of claims 1 to 8.
11. A detection model determination system, characterized in that: It comprises a processor and a memory connected to the processor, wherein the memory stores instructions executed by the processor, and the instructions enable the processor to perform operations to perform the detection model determination method according to any one of claims 1 to 8.