Defect detection model determination method and device, defect detection method and storage medium
By adding and decreasing data and annotating the image set of wafer defects, the sample number is balanced, and the problem of poor learning effect in wafer defect detection is solved, and the detection accuracy is improved.
Patent Information
- Application Number
- CN202310743217.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-06-21
- Publication Date
- 2024-12-31
AI Technical Summary
In the prior art, when using deep learning methods to detect wafer defects, there is a problem of unsatisfactory discrimination effect, especially due to the uneven sample size of different types of defects, the learning effect of deep learning models is poor.
By adding and decreasing defect image data on the target defect image set, we ensure that the number of samples of different defect types reaches consistency and balance, including image rotation, scaling, translation, color transformation, Mixup and Cutmix and other transformation processes, and generate more samples. The images are also annotated to train the deep learning model.
It improves the learning effect and detection accuracy of deep learning models, enhances the ability to identify and detect defects, and solves the problem of poor model learning results caused by uneven sample size.
Smart Images
Figure CN119228708A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of semiconductor manufacturing technology, and in particular to a defect detection model determination method, a defect detection method, a defect detection model determination device, and a storage medium. Background Art
[0002] Due to the complex production process of semiconductor wafers, there are many types of defects generated during the production process, generally up to hundreds of types. In addition, the characteristics of different types of defects are often not very clear, which makes quality inspectors feel powerless when making manual judgments. At the same time, due to the subjectivity and fatigue of people, the final judgment effect is not very good, so now deep learning-based methods are often used to automatically identify and detect wafer defects. However, in the actual application of deep learning methods to identify wafer defects, there is a problem of unsatisfactory identification effect. For example, when using a deep learning-based method to identify the defect type of certain wafer defect images, the corresponding deep learning model cannot correctly identify the defect type of the wafer defect image. Summary of the invention
[0003] In view of this, in order to solve at least some of the problems and shortcomings of the above-mentioned prior art, the present invention implements a defect detection model determination method, a defect detection model determination device, a defect detection method, a defect detection model determination device and a storage medium.
[0004] On the one hand, an embodiment of the present invention proposes a defect detection model determination method, including: acquiring multiple defect image sets; in response to the number of defect images in a target defect image set among the multiple defect image sets being not equal to a defect image number threshold, performing defect image data increase and decrease processing on the target defect image set to obtain a processed defect image set; annotating image data on the processed defect image set and defect image sets in the multiple defect image sets except the target defect image set to obtain multiple annotated defect image sets; and determining a defect detection deep learning model using the multiple annotated defect image sets to obtain a determined defect detection deep learning model.
[0005] In one embodiment of the present invention, in response to the number of defect images in a target defect image set among the multiple defect image sets being not equal to a defect image number threshold, defect image data increase and decrease processing is performed on the target defect image set to obtain a processed defect image set, including: in response to the number of defect images in the target defect image set being less than the defect image number threshold, determining and acquiring a target defect image in the target defect image set; randomly generating a first random value; in response to the first random value being less than a first image transformation threshold, performing a first image transformation processing on the target defect image to obtain a first transformed image; and classifying the first transformed image into the target defect image set.
[0006] In one embodiment of the present invention, the first image transformation process includes: at least one of image rotation transformation, image scaling transformation, image translation transformation and image color transformation.
[0007] In one embodiment of the present invention, in response to the number of defect images in a target defect image set among the multiple defect image sets being not equal to a defect image number threshold, defect image data increase and decrease processing is performed on the target defect image set to obtain a processed defect image set, including: in response to the number of defect images in the target defect image set being less than the defect image number threshold, determining and acquiring a target defect image in the target defect image set; randomly generating a first random value; in response to the first random value being greater than or equal to a first image transformation threshold, randomly generating a second random value; in response to the second random value being less than a second image transformation threshold, performing a second image transformation processing on the target defect image to obtain a second transformed image; and incorporating the second transformed image into the target defect image set.
[0008] In one embodiment of the present invention, the second image transformation process includes: at least one of image Mixup transformation and image Cutmix transformation.
[0009] In one embodiment of the present invention, in response to the number of defect images in a target defect image set among the multiple defect image sets being not equal to a defect image number threshold, performing defect image data increase and decrease processing on the target defect image set to obtain a processed defect image set, including: in response to the number of defect images in the target defect image set being greater than the defect image number threshold, determining a target defect image in the target defect image set; and deleting the target defect image.
[0010] In one embodiment of the present invention, determining the target defect image in the target defect image set includes: screening out at least two defect images whose repetition reaches a repetition threshold in the target defect image set; and taking at least one defect image of the at least two defect images as the target defect image.
[0011] On the other hand, an embodiment of the present invention provides a defect detection model determination device, which is used to execute the defect detection model determination method as described in any one of the above, and includes: a defect image set acquisition module, used to acquire multiple defect image sets; an image set data increase and decrease module, used to, in response to the number of defect images in a target defect image set among the multiple defect image sets being not equal to a defect image number threshold, perform defect image data increase and decrease processing on the target defect image set to obtain a processed defect image set; an image set data labeling module, used to perform image data labeling on the processed defect image set and the defect image sets among the multiple defect image sets except the target defect image set, to obtain multiple labeled defect image sets; and a learning model determination module, used to determine a defect detection deep learning model using the multiple labeled defect image sets to obtain a determined defect detection deep learning model.
[0012] On the other hand, an embodiment of the present invention provides a defect detection method, comprising: acquiring a defect image of a part to be tested; performing defect detection on the defect image based on a defect detection deep learning model to obtain defect information of the part to be tested, wherein the defect detection deep learning model is a determined defect detection deep learning model obtained by a defect detection model determination method as described in any one of the foregoing.
[0013] On the one hand, an embodiment of the present invention provides a storage medium which is a non-volatile memory and stores a computer program, wherein the computer program is used to execute the defect detection model determination method as described in any one of the above, and / or the defect detection method as described in the above.
[0014] At least one of the above technical solutions has the following advantages or beneficial effects:
[0015] The embodiment of the present invention obtains a processed defect image set by performing defect image data increase and decrease processing on a target defect image set whose number of defect images is not equal to a defect image number threshold, thereby making the number of defect image samples in multiple defect image sets representing different defect types consistent and balanced, avoiding the problem of poor learning effect of the deep learning model caused by the large difference in the number of defect image samples in multiple defect image sets of different types; improving the learning effect and detection effect of the defect detection model based on the deep learning model, thereby enhancing the accuracy of the deep learning model in distinguishing and detecting defects. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.
[0017] Figure 1 A schematic flow chart of a defect detection model determination method provided in the first embodiment of the present invention.
[0018] Figure 2a A schematic diagram of the statistical distribution of the number of defect samples of the multiple defect image sets.
[0019] Figure 2b It is a schematic diagram of the statistical distribution of the number of defect samples in the multiple labeled defect image sets.
[0020] Figure 3 for Figure 1 Detailed flowchart of step S200 is shown in FIG.
[0021] Figure 4 for Figure 1 Another detailed flow chart of step S200 is shown in FIG.
[0022] Figure 5 for Figure 1 Another detailed schematic flow chart of step S200 is shown in FIG.
[0023] Figure 6 for Figure 5 The detailed flow chart of step S220 is shown.
[0024] Figure 7 It is a schematic diagram of the structure of a defect detection model determination device according to the second embodiment of the present invention.
[0025] Figure 8 It is a flowchart of a defect detection method involved in the third embodiment of the present invention.
[0026] Fig. 9 FIG. 4 is a schematic diagram of the structure of a storage medium according to the fourth embodiment of the present invention. DETAILED DESCRIPTION
[0027] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0028] It should be noted that the terms "first", "second", "one end", etc. in the specification and claims of the present invention are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, products, or devices.
[0029] [First embodiment]
[0030] like Figure 1 As shown, a defect detection model determination method provided by the first embodiment of the present invention includes the steps of:
[0031] S100: taking multiple defect image sets;
[0032] S200: In response to the number of defect images in a target defect image set in the plurality of defect image sets not being equal to a defect image number threshold, performing defect image data increase and decrease processing on the target defect image set to obtain a processed defect image set;
[0033] S300: Annotating image data of the processed defect image set and defect image sets among the multiple defect image sets except the target defect image set to obtain multiple annotated defect image sets;
[0034] S400: Determine a defect detection deep learning model using the multiple annotated defect image sets to obtain a determined defect detection deep learning model.
[0035] The defect images in the defect image set come from, for example, a test piece, such as a wafer. In the actual production of semiconductor wafers, wafer defects are often concentrated in a limited number of categories. There may also be a large difference in the number of samples contained in different categories of wafer defects. Therefore, when using wafer defect images that appear in actual production to determine the deep learning model, the deep learning model will have a poor learning effect on wafer defect categories with a small number of samples. For example Figure 2aAs shown, the horizontal axis is the defect type index, and the vertical axis is the amount of data corresponding to each defect type. It can be seen that the most common defect type 1 has 5,000 samples, while the sample size of defect type 10 shown in the figure is less than one-tenth of the sample size of defect type 1. If the deep learning model is determined directly according to this data, the learning effect of the model on different types of defects will be very different. The larger the amount of data, the better the learning effect of the defect type, and vice versa. The final overall detection effect is often poor. As a result, the deep learning model has poor recognition effect on such wafer defect categories with a small number of samples. In step S100, the multiple defect image sets obtained are, for example, divided according to the type of wafer defects, such as scratch defects, center defects, ring defects, etc. corresponding to the following: Figure 2a The defect type index shown in the figure. Through the defect image data increase and decrease processing in step S200, the number of samples for the defect type with a small number of samples is increased. The defect image data increase and decrease processing can also reduce the number of samples for the defect type with a large number of samples. Therefore, the defect image data increase and decrease processing can make the number of samples of various defect types represented by the multiple defect image sets consistent, for example, Figure 2a The sample size distribution of each defect type shown becomes as follows Figure 2b The sample quantity distribution of each defect type is shown. Step S300 and step S400 are then executed. Therefore, the defect image data increase and decrease processing can avoid the situation where there are fewer samples of a certain type of defect, and thus can improve the learning effect of the deep learning model, thereby enhancing the deep learning model's ability to distinguish and detect defects. The defect detection model determines the phenomenon of uneven number of defect samples actually generated in the wafer production process. By introducing the defect image data increase and decrease processing technology to adjust this phenomenon, the determination accuracy of the subsequent deep learning model can be effectively improved, thereby improving the accuracy and speed of production line defect re-judgment when deployed and used. In addition, once the defect data is balanced and labeled, the subsequent model iteration and maintenance are very simple. The defect detection model determination method also has the advantages of being simple and effective.
[0036] Specifically, the data in the defect image data processing refers to: the number of defect images, pixel values at different positions in the defect image, etc.; which means that the defect image data processing can increase or decrease the number of defect images, and can also change the defect image. The basis for the image data annotation is the defect image set to which the defect image belongs, which means that each defect image in multiple defect image sets has been annotated, and the annotated defect images carry a mark corresponding to the defect image set to which they are located, that is, a mark of the defect type. There are many ways to annotate the image data, such as pixel marking in the defect image, marking the defect image after spatial transformation, or adding marks to the file name of the defect image and the folder name where the defect image is located, such as adding rune marks such as "category one" or "type1" to the file name.
[0037] Further, see Figure 3 , the S200 comprises the steps of:
[0038] S211: In response to the number of defect images in the target defect image set being less than the defect image number threshold, determining and acquiring a target defect image in the target defect image set.
[0039] S212: Randomly generate a first random value.
[0040] S213: In response to the first random value being less than a first image transformation threshold, performing a first image transformation process on the target defect image to obtain a first transformed image.
[0041] S214: Classify the first transformed image into the target defect image set.
[0042] The number of defect images in the target defect image set is less than the defect image number threshold, indicating that the number of samples of a certain type of defects included in the target defect image set is small. By transforming the first image, waiting for a first transformed image, and classifying the first transformed image into the target defect image set, the number of samples of a certain type of defects included in the target defect image set is increased.
[0043] Further, see Figure 4 , the S200 comprises the steps of:
[0044] S211: In response to the number of defect images in the target defect image set being less than the defect image number threshold, determining and acquiring a target defect image in the target defect image set;
[0045] S212: Randomly generate a first random value;
[0046] S215: In response to the first random value being greater than or equal to a first image transformation threshold, randomly generating a second random value;
[0047] S216: In response to the second random value being less than a second image transformation threshold, performing a second image transformation process on the target defect image to obtain a second transformed image;
[0048] S217: Adding the second transformed image to the target defect image set.
[0049] In combination with steps S211 to S217, the target defect image may undergo a first image transformation process, a second image transformation process, or remain unchanged. Specifically, steps S211 to S217 are continuously repeated in sequence until the number of defect images in the target defect image set is greater than or equal to the defect image number threshold. Specifically, the target defect image determined and obtained is, for example, a defect image.
[0050] Furthermore, the first image transformation processing includes: at least one of image rotation transformation, image scaling transformation, image translation transformation and image color transformation. For example, the defect area of the target defect image is rotated, or the size, position, and color of the defect area are changed, and the above operations can also be superimposed. Therefore, the first image transformation can generate a first transformed image different from the target defect image to increase the number of samples of the defect type corresponding to the target defect image. In addition, the defect type of the defect image before and after the first image transformation processing is the same.
[0051] Furthermore, the second image transformation processing includes: at least one of image mixing (Mixup) transformation and image mixing (Cutmix) transformation. The image mixing transformation is, for example, extracting a defect image from the plurality of defect image sets as a reference image, and mixing the reference image with the target defect image, and the pixel value of each pixel point of the mixed image is between the pixel value of the reference image and the pixel value of the target defect image at the pixel point. The image mixing transformation is, for example, intercepting a part of the reference image, deleting the image of the target defect image in the same area, and then moving the intercepted part of the reference image into the image deletion area of the target defect image to generate a new defect image. Therefore, the second image transformation can also increase the number of samples of the defect type corresponding to the target defect image. The defect image samples obtained by the image mixing transformation can enhance the generalization ability of the deep learning model and reduce the overfitting phenomenon. The image mixing transformation will not allow non-information pixels to appear in the sample, and requires the model to be able to locate the location of the defect feature. Compared with the image mixing transformation, the mixed image will not be unnatural.
[0052] Further, see Figure 5 , the S200 comprises the steps of:
[0053] S220: In response to the number of defect images in the target defect image set being greater than the defect image number threshold, determining a target defect image in the target defect image set;
[0054] S230: deleting the target defect image;
[0055] Step S220 and step S230 are, for example, executed in a loop until the number of defective images in the target defective image set is less than or equal to the defective image number threshold.
[0056] Further, see Figure 6 , the step S220 comprises the steps of:
[0057] S221: Screening out at least two defect images whose repetition reaches a repetition threshold from the target defect image set;
[0058] S222: Using at least one defect image among the at least two defect images as the target defect image.
[0059] Specifically, the deletion of the target defect image in step S230 includes, for example, randomly deleting the target defect image or preferentially deleting the target defect image with a higher degree of repetition. The implementation of S221 is, for example, to select a defect image arbitrarily in the target defect image set, and compare the defect image with other defect images in the target defect image set to determine the defect image that reaches the repetition threshold. By repeating this process, all defect images in the target defect image set are found. The repetition is, for example, the size of the area of similar defects according to the pixel distribution of the image. If the defect size of the other defect images is compared with the defect area of the one defect image, and the defect area is 90% to 110% of the defect area of the one defect image, then the other defect images are judged to have a repetition that reaches the repetition threshold. The repetition can also be, for example, the location of the defects of two similar defect images, or it can be to prioritize the size of the area and then judge the location of the defects. The repetition can be set according to actual needs. Specifically, in S222, it is preferred to put one defect image back into the target defect image set, and the remaining defect images are used as the target defect images. This reduces the deleting of the target defect image and weakens the diversity of samples in the target defect image set.
[0060] To facilitate understanding of the present invention, the preferred implementation of each step of the defect detection model determination method according to an embodiment of the present invention will be described in detail below.
[0061] like Figure 1As shown, multiple defect image sets are obtained through S100. The multiple defect image sets are divided according to defect types, that is, each defect image set corresponds to a defect type. After obtaining the multiple defect image sets, a target defect image set is obtained through S200 judgment response, and the target defect image set is processed by defect image data addition and subtraction to obtain a processed defect image set.
[0062] Specifically, S200 performs a judgment response on each defect image set in the plurality of defect image sets. Before executing S200, a defect image quantity threshold value needs to be set or generated by the machine. The defect image quantity threshold value can be determined, for example, based on the distribution of the defect image quantities of the plurality of defect image sets. For example, the defect image quantity threshold value is set at Figure 2a The middle level of the number of defective images in the plurality of defective image sets is shown, such as 2000. One defective image set in the plurality of defective image sets is confirmed as a target defective image set. If the number of defective images in the target defective image set is not equal to the defective image number threshold, defective image data increase or decrease processing is performed on the target defective image set.
[0063] Furthermore, if the number of defective images in the target defective image set is less than the defective image number threshold, then Figure 3 As shown, execute S211 to determine and obtain the target defect image in the target defect image set; and execute S212 to randomly generate a first random value; and obtain a first image transformation threshold, the first image transformation threshold can be set manually, or can be generated by a computer or other device. Specifically, the value range of the first image transformation threshold and the value range of the first random value are preferably the same, for example, both are within a closed interval of 0 to 1; then execute S213 to perform a first image transformation process on the target defect image to obtain a first transformed image.
[0064] Furthermore, the first image transformation processing includes: at least one of image rotation transformation, image scaling transformation, image translation transformation and image color transformation. For example, the defect area of the target defect image is rotated, or the size, position and color of the defect area are changed, and the above operations can also be superimposed. Here, the target defect image is a grayscale image with a resolution of 512x512, and the target defect image is subjected to the image translation transformation or the image rotation transformation. Therefore, the information of each pixel point of the target defect image can be expressed by a three-dimensional array written as (x, y, z), and the x and y express the position of the pixel point on the target defect image, and the value range is, for example, [-256, 256]. The z expresses the grayscale value at the pixel point, and the grayscale value range is [0, 255]. The larger the grayscale value, the closer the pixel point is to white. If the target defect image is subjected to an image translation transformation, the target defect image is first copied and a new image is generated.
[0065] Then select the defect position of the new image, for example, the defect position is expressed by the position of the pixel point on the new image, which is (x, y), and the value range of x and y is [-256, 0). Then move the pixel points in the range of (x, y), for example, to (x1, y1), and the value range of x1 and y1 is (0, 256). The mapping relationship between (x, y) and (x1, y1) is, for example, x+256=x1, y+256=y1, that is, the position coordinates of the pixel point at (x, y) before the image translation transformation begins are added 256 respectively, and moved to the position of (x1, y1), and at the same time, the pixel point at (x1, y1) before the image translation transformation begins is moved to the position of (x1, y1). The pixel coordinates are subtracted by 256 and moved to the position (x, y). In order to realize the image translation transformation, a blank image with a resolution of 512x512 can also be created, the pixel at the position (x1, y1) is moved to the corresponding position of the blank image, and the pixel at (x, y) is moved to (x1, y1) in the new image, and then the pixel at (x1, y1) in the blank image is moved to the position (x, y) of the new image.
[0066] If the new image is subjected to an image rotation transformation, as described above, the expression of the information of each pixel point of the new image can be expressed by a three-dimensional array written as (x, y, z) as shown in the above image translation transformation. First, the defect position of the new image is selected. For example, the defect position is expressed by the position of the pixel point on the new image, which is (x, y), and the value range of x and y is [-256, 0). Then the pixels in the range of (x, y) are rotated. For example, rotate to (x1, y1), and the value range of x1 and y1 is (0, 256). The rotation matrix of the image rotation transformation in this example is:
[0067]
[0068] The transformation relationship between (x, y) and (x1, y1) is:
[0069]
[0070] The image translation transformation and the image rotation transformation can be performed on the same new image. The defect position of the new image can be determined manually or by a machine device. For example, if a ring defect exists on the new image, the position of the ring defect on the new image can be selected manually. If the defect type on the new image is a random defect, the machine device can randomly determine the location of the defect. How to determine the defect position of the new image can be determined by the actual situation of the defect image, and there is no limitation here.
[0071] After the first image transformation is completed on the target defect image, that is, the first image transformation is completed on the new image and the first transformed image is obtained, step S214 is performed to classify the first transformed image into the target defect image set. At this point, the number of defect images in the target defect image set has been increased, and because of the first image transformation operation, the sample diversity in the target defect image set has also been enhanced.
[0072] If the first random value is greater than or equal to the first image transformation threshold, the target defect image will not be subjected to the first image transformation. Figure 4Step S215 shown in the figure, and obtaining or randomly generating a second image transformation threshold, the second random value obtained or randomly generated has the same value interval as the second image transformation threshold, for example, both are closed intervals from 0 to 1. If the second random value is greater than or equal to the second image transformation threshold, the first image transformation is not performed on the target defect image, and the second image transformation is not performed on the target defect image. The target defect image is then included in the target defect image set. If the second random value is less than the second image transformation threshold, the following is performed: Figure 4 Step S216 shown.
[0073] Furthermore, the second image transformation includes: at least one of an image mixing (Mixup) transformation and an image cutting (Cutmix) transformation. An implementation of the image mixing transformation is as follows. A defect image is extracted from the plurality of defect image sets as a reference image. The target defect image and the reference image are, for example, grayscale images. The description of the information of each pixel in the grayscale image is described as above, and is described by a three-dimensional array, such as (x, y, z). In the image mixing process, the pixel value of the mixed image at each pixel is between the pixel values of the reference image and the target defect image at the pixel. For example, z in (x, y, z) is a grayscale value, that is, a pixel value of the defect image. Let the three-dimensional array of the reference image be written as (x1, y1, z1); let the three-dimensional array of the target defect image be written as (x2, y2, z2); let the three-dimensional array of the mixed image be written as (x3, y3, z3). Then the relationship between the reference image, the target defect image and the mixed image at each corresponding pixel is, for example, as shown in the following formula:
[0074] x1=x2=x3;
[0075] y1=y2=y3;
[0076] z3=λz1+(1-λ)z2;
[0077] The λ is a number between 0 and 1, which can be set according to actual conditions.
[0078] The image blending transformation is, for example, to intercept a part of the reference image, delete the image of the target defect image in the same area, and then move the intercepted part of the reference image into the image deletion area of the target defect image to generate a new defect image. An implementation of the image blending transformation is as follows. Extract a defect image from the plurality of defect image sets as a reference image. The target defect image and the reference image are, for example, grayscale images. The description of the information of each pixel in the grayscale image is described as described above, and is described by a three-dimensional array, such as (x, y, z). In the image blending process, the pixel value of each pixel of the blended image is between the pixel value of the reference image and the pixel value of the target defect image at the pixel. For example, z in (x, y, z) is a grayscale value, that is, a pixel value of the defect image. Let the three-dimensional array of the reference image be written as (x1, y1, z1); let the three-dimensional array of the target defect image be written as (x2, y2, z2); let the three-dimensional array of the blended image be written as (x3, y3, z3). Then the relationship between the reference image, the target defect image and the mixed image at each corresponding pixel point is, for example, as shown in the following formula:
[0079] x1=x2=x3;
[0080] y1=y2=y3;
[0081] z3=z1,
[0082] When -256≤x3<0 and -256≤y3<0;
[0083] When either or both of the conditions -256≤x3<0 and -256≤y3<0 are not met,
[0084] z3=z2.
[0085] The above-mentioned implementation example of image blending transformation is only a possibility. The setting of the numerical range of the above-mentioned one or two conditions that x3 and y3 need to satisfy can be set according to actual conditions. For example, the setting of the numerical range is performed according to the location of the defect.
[0086] Therefore, the image blending transformation and the image mixing transformation included in the second image transformation can both put two different defects into the same image. In addition, the image blending transformation and the image mixing transformation are not limited to executing only one, and both can be performed on the target defect image.
[0087] This embodiment does not limit the target defect image to a grayscale image. The target defect image may also be a color image such as an RGB format. In this case, the dimension describing the pixel points in the target defect image is increased by two, and the increased two dimensions are used to express the color. The method of performing the first image processing or the second image processing on the color target defect image refers to the case where the target defect image is a grayscale image.
[0088] After completing the second image transformation, the mixed image or the blended image is included in the target defect image set as the second transformed image, that is, the following is performed: Figure 4 Step S217 shown.
[0089] When the number of defect images in the target defect image set is greater than the defect image number threshold, a target defect image is determined in the target defect image set, that is, the following is performed: Figure 5 Step S220 is shown. Then, the following steps are performed: Figure 6 The step S221 shown is performed, and then step S222 is performed. The implementation of S221 is, for example, to arbitrarily select a defect image in the target defect image set, and compare the one defect image with other defect images in the target defect image set to determine the defect image that reaches the repeatability threshold. By repeating this process, all defect images in the target defect image set are found. The repeatability is, for example, the size of the area of similar defects based on the pixel distribution of the image. If the defect size of the other defect images is compared with the one defect image, and the defect area is 90% to 110% of the defect area of the one defect image, then the other defect images are judged to have a repeatability reaching the repeatability threshold. The repeatability can also be, for example, the location of the defects of two similar defect images, or it can be to prioritize the size of the area and then judge the location of the defect. The repeatability can be set according to actual needs. Specifically, in S222, it is preferred to put one defect image back into the target defect image set, and the remaining defect images are used as the target defect images.
[0090] When each of the multiple defect image sets meets the response condition of step S200 and has undergone the defect image data increase and decrease processing, the above step S300 is performed to annotate the image data of the processed defect image set and the defect image sets other than the target defect image set in the multiple defect image sets to obtain multiple annotated defect image sets, and step S400 is performed to determine the defect detection deep learning model using the multiple annotated defect image sets to obtain the determined defect detection deep learning model. The above defect detection deep learning model can be used to distinguish the defect type of the defect image.
[0091] [Second embodiment]
[0092] See also Figure 7 The second embodiment of the present invention provides a defect detection model determination device 10. The defect detection model determination device 10 is used to perform the defect detection model determination method as described in the first embodiment. The defect detection model determination device 10 includes: a defect image set acquisition module 100, an image set data increase and decrease module 200, an image set data annotation module 300 and a learning model determination module 400.
[0093] Specifically, the defect image set acquisition module 100 is used to acquire multiple defect image sets; the image set data increase and decrease module 200 is used to, in response to the number of defect images in a target defect image set among the multiple defect image sets being not equal to a defect image number threshold, perform defect image data increase and decrease processing on the target defect image set to obtain a processed defect image set; the image set data annotation module 300 is used to perform image data annotation on the processed defect image set and the defect image sets among the multiple defect image sets except the target defect image set to obtain multiple annotated defect image sets; and the learning model determination module 400 is used to determine a defect detection deep learning model using the multiple annotated defect image sets to obtain a determined defect detection deep learning model.
[0094] The specific working process and technical effects of each module in the defect detection model determination device described in this embodiment can be found in the relevant step description of the first embodiment mentioned above, and will not be repeated here.
[0095] [Third embodiment]
[0096] See also Figure 8 The third embodiment of the present invention provides a defect detection method. The defect detection method comprises the steps of:
[0097] S500: Acquire a defect image of the test piece;
[0098] S600: Perform defect detection on the defect image based on a defect detection deep learning model to obtain defect information of the test piece, wherein the defect detection deep learning model is a determined defect detection deep learning model obtained by the defect detection model determination method described in the first embodiment.
[0099] The device to be tested here is, for example, the wafer in the first embodiment. The defect detection deep learning model based on the defect detection method is determined by the defect detection model determination method described in the first embodiment. Therefore, the defect detection method provided by the embodiment of the present invention has better accuracy in defect detection and discrimination.
[0100] [Fourth embodiment]
[0101] See also Fig. 9 A storage medium 500 provided in a fourth embodiment of the present invention is a non-volatile memory and stores a computer program. The computer program is used to execute the defect detection model determination method described in the first embodiment and / or the defect detection method described in the third embodiment. The storage medium 500 is, for example, a non-volatile memory, such as: magnetic media (such as hard disks, floppy disks and tapes), optical media (such as CDROM disks and DVDs), magneto-optical media (such as optical disks) and hardware devices specially constructed for storing and executing computer executable instructions (such as read-only memory (ROM), random access memory (RAM), flash memory, etc.). The storage medium 500 can be executed by one or more processors or processing devices to execute the computer program.
[0102] In addition, it can be understood that the aforementioned embodiments are only exemplary illustrations of the present invention. Under the premise that the technical features do not conflict, are not contradictory, and do not violate the purpose of the present invention, the technical solutions of the various embodiments can be arbitrarily combined and used in combination.
[0103] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.
[0104] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.
[0105] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.
[0106] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A defect detection model determination method, characterized in that: include: Acquire multiple defect image sets; In response to the number of defect images in a target defect image set among the multiple defect image sets not being equal to a defect image number threshold, performing defect image data increase and decrease processing on the target defect image set to obtain a processed defect image set; Annotating image data of the processed defect image set and defect image sets in the multiple defect image sets except the target defect image set to obtain multiple annotated defect image sets; as well as The defect detection deep learning model is determined using the multiple annotated defect image sets to obtain a determined defect detection deep learning model.
2. The defect detection model determination method according to claim 1, characterized in that: In response to the number of defect images in the target defect image set among the multiple defect image sets not being equal to the defect image number threshold, performing defect image data increase and decrease processing on the target defect image set to obtain a processed defect image set, comprising: In response to the number of defect images in the target defect image set being less than the defect image number threshold, determining and acquiring a target defect image in the target defect image set; Randomly generate a first random value; In response to the first random value being less than a first image transformation threshold, performing a first image transformation process on the target defect image to obtain a first transformed image; and The first transformed image is included in the target defect image set.
3. The defect detection model determination method according to claim 2, characterized in that: The first image transformation processing includes: at least one of image rotation transformation, image scaling transformation, image translation transformation and image color transformation.
4. The defect detection model determination method according to claim 1, characterized in that: In response to the number of defect images in the target defect image set among the multiple defect image sets not being equal to the defect image number threshold, performing defect image data increase and decrease processing on the target defect image set to obtain a processed defect image set, comprising: In response to the number of defect images in the target defect image set being less than the defect image number threshold, determining and acquiring a target defect image in the target defect image set; Randomly generate a first random value; In response to the first random value being greater than or equal to a first image transformation threshold, randomly generating a second random value; In response to the second random value being less than a second image transformation threshold, performing a second image transformation process on the target defect image to obtain a second transformed image; and The second transformed image is included in the target defect image set.
5. The defect detection model determination method according to claim 4, characterized in that: The second image transformation process includes: at least one of image Mixup transformation and image Cutmix transformation.
6. The defect detection model determination method according to claim 1, characterized in that: In response to the number of defect images in the target defect image set among the multiple defect image sets not being equal to the defect image number threshold, performing defect image data increase and decrease processing on the target defect image set to obtain a processed defect image set, comprising: In response to the number of defect images in the target defect image set being greater than the defect image number threshold, determining a target defect image in the target defect image set; and The target defect image is deleted.
7. The defect detection model determination method according to claim 6, characterized in that: Determining the target defect image in the target defect image set comprises: Screening out at least two defect images whose repetition reaches a repetition threshold from the target defect image set; At least one defect image among the at least two defect images is used as the target defect image.
8. A defect detection model determination device, characterized in that: Used to execute the defect detection model determination method according to any one of claims 1 to 7 and comprising: A defect image set acquisition module, used for acquiring multiple defect image sets; An image set data increasing and decreasing module, configured to, in response to the number of defect images in a target defect image set in the plurality of defect image sets not being equal to a defect image number threshold, perform defect image data increasing and decreasing processing on the target defect image set to obtain a processed defect image set; an image set data annotation module, configured to annotate image data of the processed defect image set and defect image sets among the multiple defect image sets except the target defect image set, to obtain multiple annotated defect image sets; and The learning model determination module is used to determine the defect detection deep learning model using the multiple annotated defect image sets to obtain a determined defect detection deep learning model.
9. A defect detection method, characterized in that: include: Acquire defect images of the test piece; Defect detection is performed on the defect image based on a defect detection deep learning model to obtain defect information of the test piece, wherein the defect detection deep learning model is a determined defect detection deep learning model obtained by the defect detection model determination method according to any one of claims 1 to 7.
10. A storage medium, which is a non-volatile memory and stores a computer program, characterized in that: The computer program is used to execute the defect detection model determination method according to any one of claims 1 to 7, and / or the defect detection method according to claim 9.