Method and System for Detecting Wafer Defects Based on Attention Pyramid Change Detection

Abstract

The present application provides a method and system for detecting wafer defects based on attention pyramid change detection. The method includes: preprocessing a pre-shot wafer image and annotating the defective part of the preprocessed wafer image; constructing a feature extraction network, inputting the defective part into the feature extraction network to obtain an output feature map; optimizing the deep features in the output feature map; generating a dynamic change prior matrix according to the optimized deep features; jointly optimizing based on the dynamic change prior matrix and the features of the output feature map, and outputting an optimized output feature map; training the feature extraction network according to the optimized output feature map and the annotated defective part; inputting the wafer image to be detected into the trained feature extraction network, and outputting the wafer defect detection result. The present application has high robustness and strong generalization ability, makes full use of the information of the template, and can achieve fast and accurate detection even for defects that have not participated in training.

Description

Technical Field

[0001] This application relates to the field of industrial vision inspection technology, and particularly to a method and system for detecting wafer defects by change detection of an attention pyramid based on dynamic prior enhancement. Background Art

[0002] Existing wafer defect detection technologies mainly rely on image processing or object detection methods. Among them, the image processing-based method realizes defect recognition by extracting templates and the features of the extracted templates and input images; the object detection-based method identifies the positions of defects by training a model after annotating existing defect data.

[0003] Due to factors such as noise and illumination, the image processing-based method has a high false detection rate, and eliminating these effects through other preprocessing methods usually involves complex algorithm design; while the object detection-based method has relatively high robustness, but it overly relies on the annotation of existing defects and ignores the characteristics of the wafer itself, and cannot detect defects that have not been trained. Summary of the Invention

[0004] In view of this, the purpose of this application is to propose a method and system for detecting wafer defects by change detection of an attention pyramid based on dynamic prior enhancement, and this application can specifically solve existing problems.

[0005] Based on the above purpose, this application proposes a method for detecting wafer defects by change detection of an attention pyramid based on dynamic prior enhancement, including:

[0006] Preprocess the pre-shot wafer image and label the defective parts of the preprocessed wafer image;

[0007] Construct a feature extraction network, input the defective parts into the feature extraction network, and obtain an output feature map;

[0008] Optimize the deep features in the output feature map;

[0009] Generate a dynamic change prior matrix according to the optimized deep features;

[0010] Based on the dynamic change prior matrix and the features of the output feature map, perform joint optimization and output an optimized output feature map;

[0011] Train the feature extraction network according to the optimized output feature map and the labeled defective parts;

[0012] Input the wafer image to be detected into the trained feature extraction network and output the wafer defect detection result.

[0013] For the above purposes, the present application also proposes a system for detecting wafer defects based on an attention pyramid with dynamic prior enhancement, including:

[0014] A data processing module, configured to preprocess a pre-shot wafer image and label the defective parts of the preprocessed wafer image;

[0015] A network construction module, configured to construct a feature extraction network, input the defective parts into the feature extraction network, and obtain an output feature map;

[0016] A deep feature optimization module, configured to optimize the deep features in the output feature map;

[0017] A dynamic prior module, configured to generate a dynamic change prior matrix according to the optimized deep features;

[0018] A joint optimization module, configured to perform joint optimization based on the dynamic change prior matrix and the features of the output feature map, and output an optimized output feature map;

[0019] A training module, configured to train the feature extraction network according to the optimized output feature map and the labeled defective parts;

[0020] A detection module, configured to input a wafer image to be detected into the trained feature extraction network and output a wafer defect detection result.

[0021] Generally speaking, the advantages of the present application and the experience brought to users are as follows:

[0022] 1. High robustness. Through simulation training under various lighting and noise conditions, high-precision detection of defects under complex conditions can be achieved.

[0023] 2. Strong generalization ability. By making full use of the information of the template, fast and accurate detection can be achieved even for defects that have not participated in the training. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In the drawings, unless otherwise specified, the same reference numerals throughout the several views denote the same or similar components or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings only depict some embodiments disclosed according to the present application and should not be regarded as limiting the scope of the present application.

[0025] Figure 1 FIG. shows a flowchart of a method for detecting wafer defects based on an attention pyramid with dynamic prior enhancement according to an embodiment of the present application.

[0026] Figure 2The block diagram of a system for detecting wafer defects based on dynamic prior-enhanced attention pyramid change detection according to an embodiment of the present application is shown.

[0027] Figure 3 The schematic structural diagram of an electronic device provided by an embodiment of the present application is shown.

[0028] Figure 4 The schematic diagram of a storage medium provided by an embodiment of the present application is shown. Detailed implementation manners

[0029] The present application will be further described in detail below in conjunction with the drawings and embodiments. It can be understood that the specific embodiments described herein are only used to explain the related invention, rather than limiting the invention. In addition, it should be noted that for the sake of convenience of description, only the parts related to the invention are shown in the drawings.

[0030] It should be noted that, without conflict, the embodiments in the present application and the features in the embodiments can be combined with each other. The present application will be described in detail below with reference to the drawings and embodiments.

[0031] The present invention proposes a method for detecting wafer defects based on dynamic prior-enhanced attention pyramid change detection. The detection of defects is achieved by comparing the features of the template image and the defect image, which can make full use of the defect and its own structural features to accurately identify any defect. Specifically, the method includes the following steps.

[0032] S1: Data preparation

[0033] The wafer images are scanned into grayscale images, and the median value of each pixel of multiple wafer images is taken to obtain the median image I ref as the template image.

[0034] These images are cropped and normalized, and the size is adjusted to 256×256 pixels to meet the model input requirements. And the defective parts of some of these images are labeled using labeling software, represented by (X A , X B ), and the binary defects labeled are represented by Y, where X represents the image set, A represents the defective image, and B represents the corresponding slice of the template image.

[0035] S2: Construct a feature extraction network

[0036] Use resnet50 as the feature extraction network, and input (X A , X B ) into the feature extraction network respectively, and take the output feature maps of the 2-5 stages , i ∈ [2, 5].

[0037] S3: Deep Feature Optimization

[0038] To improve the recognition ability of wafer defects, cosine similarity is defined here and cosine loss is calculated. The cosine loss function is used to optimize the deep features to minimize the directional differences between features and improve the accuracy of feature representation. Here, the cosine loss function optimizes the distinguishability between the regions belonging to defects and the regions not belonging to defects in the extracted deep features by affecting the gradients of the feature extraction network, thereby enhancing the performance of the network. The cosine similarity can be expressed as:

[0039] ,

[0040] The cosine loss function can be expressed as:

[0041] ,

[0042] where N represents the number of defect-free regions, j represents the pixel coordinates of the defect-free regions, C represents the number of defect regions, k represents the pixel coordinates of the defect regions, represents the feature vector of the pixel corresponding to the defect-free region in represents the feature vector of the pixel corresponding to the defect-free region in represents the feature vector of the pixel corresponding to the defect region in represents the feature vector of the pixel corresponding to the defect region in

[0043] In the way that the cosine loss function affects the gradients of the feature extraction network, the present invention uses the gradient descent method to calculate this process:

[0044] ,

[0045] where represents the feature extraction network parameters to be optimized, represents the optimized feature extraction network parameters, is the learning rate, which is taken as 0.001 here, represents the partial derivative of the cosine loss function with respect to the feature extraction network parameters, indicating the influence of the change of the cosine loss function on the weights.

[0046] S4: Generate Dynamic Prior

[0047] Normalize to ensure that the value range is between 0 and 1, and obtain the dynamic prior matrix W,

[0048] ,

[0049] Utilize this dynamically changing prior matrix to reflect the importance of different regions and guide subsequent multi-scale feature fusion. To make the size of the dynamically changing prior matrix adapt to the size of feature maps at different stages, scale it to the corresponding size, which is represented by :

[0050] ,

[0051] Here, resize represents the bilinear interpolation method.

[0052] S5: Joint Optimization of Dynamic Prior and Multi-scale Features

[0053] Concatenate the features of along the channel dimension to obtain :

[0054] ,

[0055] Use the dynamic prior to guide the spatial attention features of the feature maps at each stage :

[0056] ,

[0057] Introduce the feature pyramid structure, which can enhance the network's perception ability of small defects on the wafer. Integrate it with the guided attention extremely, then each layer of features in the pyramid can be expressed as :

[0058] ,

[0059] where Conv represents convolution and upsample represents the upsampling operation, which is used to magnify the deep features to the same size as the shallow features upward using bilinear interpolation.

[0060] S6: Output Feature Map

[0061] Upsample and convolve the final to obtain the final output feature map , which represents the probability that each pixel belongs to the defect area:

[0062] ,

[0063] The structure of

[0064]

[0065] After obtaining , use the cross-entropy loss function with it and the annotation Y Perform calculations for supervising the training of the network:

[0066] ,

[0067] Thus, the total loss function of the network can be expressed as:

[0068] ,

[0069] where is the weight of the cosine loss function, which is set to 0.1 here.

[0070] S7: Defect detection

[0071] After the network training is completed, the image to be detected can be input into the network to obtain the detection result of the image Then, use the threshold t to filter it. The area greater than the threshold is the defect area, and the area less than the threshold is the background area, thus completing the whole process of defect detection.

[0072] The application embodiment provides a system for detecting wafer defects based on dynamic prior enhanced attention pyramid change detection. This system is used to execute the method for detecting wafer defects based on dynamic prior enhanced attention pyramid change detection described in the above embodiment, as Figure 2 shown. This system includes:

[0073] A data processing module 501, configured to preprocess the pre-shot wafer image and label the defective part of the preprocessed wafer image;

[0074] A network construction module 502, configured to construct a feature extraction network, input the defective part into the feature extraction network, and obtain an output feature map;

[0075] A deep feature optimization module 503, configured to optimize the deep features in the output feature map;

[0076] A dynamic prior module 504, configured to generate a dynamic change prior matrix according to the optimized deep features;

[0077] A joint optimization module 505, configured to perform joint optimization based on the dynamic change prior matrix and the features of the output feature map, and output an optimized output feature map;

[0078] A training module 506, configured to train the feature extraction network according to the optimized output feature map and the labeled defective part;

[0079] A detection module 507, configured to input the wafer image to be detected into the trained feature extraction network and output the wafer defect detection result.

[0080] The system for detecting wafer defects based on dynamic prior enhancement of attention pyramid provided by the above embodiments of the present application and the method for detecting wafer defects based on dynamic prior enhancement of attention pyramid provided by the embodiments of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the application programs stored therein.

[0081] The embodiments of the present application also provide an electronic device corresponding to the method for detecting wafer defects based on dynamic prior enhancement of attention pyramid provided by the foregoing embodiments to execute the method for detecting wafer defects based on dynamic prior enhancement of attention pyramid. The embodiments of the present application do not make any limitations.

[0082] Please refer to Figure 3 , which shows a schematic diagram of an electronic device provided by some embodiments of the present application. As Figure 3 shown, the electronic device 20 includes: a processor 200, a memory 201, a bus 202, and a communication interface 203. The processor 200, the communication interface 203, and the memory 201 are connected through the bus 202. A computer program that can run on the processor 200 is stored in the memory 201. When the processor 200 runs the computer program, it executes the method for detecting wafer defects based on dynamic prior enhancement of attention pyramid provided by any of the foregoing embodiments of the present application.

[0083] Among them, the memory 201 may include a high-speed random access memory (RAM: Random Access Memory), and may also include a non-volatile memory, such as at least one disk memory. Through at least one communication interface 203 (which can be wired or wireless), a communication connection is realized between this system network element and at least one other network element, and the Internet, wide area network, local area network, metropolitan area network, etc. can be used.

[0084] The bus 202 may be an ISA bus, a PCI bus, an EISA bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. Among them, the memory 201 is used to store a program. After receiving an execution instruction, the processor 200 executes the program. The method for detecting wafer defects based on dynamic prior enhancement of attention pyramid disclosed in any of the foregoing embodiments of the present application can be applied to the processor 200 or implemented by the processor 200.

[0085] The processor 200 may be an integrated circuit chip with the ability to process signals. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in the processor 200 or the instructions in the form of software. The above-mentioned processor 200 may be a general-purpose processor, including a central processing unit (CPU for short), a network processor (NP for short), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. It can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in combination with the embodiments of the present application can be directly embodied as being executed and completed by the hardware decoding processor, or executed and completed by the combination of the hardware and software modules in the decoding processor. The software module may be located in a mature storage medium in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. This storage medium is located in the memory 201, and the processor 200 reads the information in the memory 201 and combines its hardware to complete the steps of the above method.

[0086] The electronic device provided by the embodiments of the present application and the method for detecting wafer defects based on dynamic prior enhancement of the attention pyramid provided by the embodiments of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by them.

[0087] The embodiments of the present application also provide a computer-readable storage medium corresponding to the method for detecting wafer defects based on dynamic prior enhancement of the attention pyramid provided by the foregoing embodiments. Please refer to Figure 4 , which shows that the computer-readable storage medium is an optical disc 30, on which a computer program (i.e., a program product) is stored. When the computer program is run by a processor, it will execute the method for detecting wafer defects based on dynamic prior enhancement of the attention pyramid provided by any of the foregoing embodiments.

[0088] It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical and magnetic storage media, which will not be elaborated here one by one.

[0089] The computer-readable storage medium provided in the above embodiments of the present application and the method for detecting wafer defects based on dynamic prior-enhanced attention pyramid change detection provided in the embodiments of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run, or implemented by the application programs stored therein.

[0090] It should be noted that:

[0091] The algorithms and displays provided herein are not inherently related to any particular computer, virtual system, or other device. Various general-purpose systems can also be used in conjunction with the teachings provided herein. The structure required to construct such systems will be apparent from the above description. In addition, the present application is not directed to any particular programming language. It should be understood that the content of the present application described herein can be implemented using various programming languages, and the description of the specific language above is for the purpose of disclosing the best mode of the present application.

[0092] In the specification provided herein, a large number of specific details are set forth. However, it can be understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known methods, structures, and technologies have not been shown in detail so as not to obscure the understanding of this specification.

[0093] Similarly, it should be understood that, in order to streamline the present application and assist in understanding one or more of the various inventive aspects, in the above description of the exemplary embodiments of the present application, the various features of the present application are sometimes grouped together into a single embodiment, figure, or description thereof. However, the disclosed method should not be construed as reflecting an intention that the claimed subject matter of the present application requires more features than are expressly recited in each claim. Rather, as reflected in the following claims, the inventive aspects lie in less than all of the features of the preceding disclosed single embodiment. Thus, the claims following the detailed description are hereby expressly incorporated into the detailed description, where each claim stands on its own as a separate embodiment of the present application.

[0094] Those skilled in the art can understand that the modules in the devices in the embodiments can be adaptively changed and set in one or more devices different from the embodiments. The modules or units or components in the embodiments can be combined into one module or unit or component, and in addition, they can be divided into multiple sub-modules or sub-units or sub-components. Except that at least some of such features and / or processes or units are mutually exclusive, any combination can be used to combine all the features disclosed in this specification (including the accompanying claims, abstract and drawings) and all the processes or units of any method or device so disclosed. Unless otherwise explicitly stated, each feature disclosed in this specification (including the accompanying claims, abstract and drawings) can be replaced by an alternative feature that provides the same, equivalent or similar purpose.

[0095] In addition, those skilled in the art can understand that although some of the embodiments described herein include certain features included in other embodiments rather than other features, the combination of the features of different embodiments means that it is within the scope of this application and forms different embodiments. For example, in the following claims, any one of the claimed embodiments can be used in any combination.

[0096] Each component embodiment of the present application can be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. Those skilled in the art should understand that in practice, a microprocessor or a digital signal processor (DSP) can be used to implement some or all of the functions of some or all of the components in the virtual machine creation system according to the embodiments of the present application. The present application can also be implemented as a device or system program (such as a computer program and a computer program product) for executing part or all of the methods described herein. Such a program implementing the present application can be stored on a computer-readable medium, or can be in the form of one or more signals. Such signals can be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0097] It should be noted that the above embodiments are illustrative of the present application rather than restrictive of the present application, and those skilled in the art can design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word "comprising" does not exclude the presence of elements or steps not listed in the claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present application can be implemented by means of hardware including several different elements and by means of a suitably programmed computer. In the unit claims listing several systems, several of these systems may be embodied by the same item of hardware. The use of the words first, second, and third, etc. does not denote any order. These words may be interpreted as names.

[0098] As described above, the above are only specific embodiments of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present application can easily think of various changes or substitutions, and these should all be covered within the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the said claims.

Claims

1. A method for detecting wafer defects based on attention pyramid changes, characterized in that: include: Preprocessing the pre-shot wafer image and marking defective parts of the preprocessed wafer image; Constructing a feature extraction network, inputting the defect part into the feature extraction network, and obtaining an output feature map; Optimizing the deep features in the output feature map; Generate a dynamically changing prior matrix based on the optimized deep features; Perform joint optimization based on the features of the dynamically changing prior matrix and the output feature map, and output an optimized output feature map; Training the feature extraction network according to the optimized output feature map and the marked defect part; Inputting the wafer image to be inspected into the trained feature extraction network, and outputting the wafer defect detection result; The preprocessing of the pre-shot wafer image and marking of defective parts of the preprocessed wafer image include: Scanning the wafer image into a grayscale image, and taking the median of each pixel of multiple wafer images to obtain a median image as a template image; The plurality of wafer images are cropped and normalized, and the size is adjusted to 256×256 pixels to meet the model input requirements; The defective parts of the plurality of wafer images are marked with (X A , X B ), and the marked binary defects are represented by Y, where X represents the image set, A represents the defect image, and B represents the corresponding slice of the template image; The step of constructing a feature extraction network, inputting the defect portion into the feature extraction network, and obtaining an output feature map comprises: Use resnet50 as the feature extraction network, and (X A , X B ) are input into the feature extraction network respectively, and the output feature maps of stages 2-5 are taken. , i∈[2,5]; The optimizing the deep features in the output feature map includes: Define cosine similarity and calculate cosine loss, and use cosine loss function to analyze deep features The optimization is performed to minimize the directional differences between features. The cosine loss function optimizes the distinction between defective areas and non-defective areas in the extracted deep features by affecting the gradient of the feature extraction network. The cosine similarity is expressed as: ， The cosine loss function is expressed as: ， Where N represents the number of non-defective areas, j represents the pixel coordinates of non-defective areas, C represents the number of defective areas, and k represents the pixel coordinates of defective areas. represent The feature vector of the pixels corresponding to the defect-free area in represent The feature vector of the pixels corresponding to the defect-free area, represent The feature vector of the pixel corresponding to the defect area in represent The feature vector of the pixel corresponding to the defect area in .

2. The method according to claim 1, characterized in that The method of generating a dynamically changing prior matrix based on the optimized deep features includes: Will Normalize to ensure that the value range is between 0 and 1, and obtain the dynamically changing prior matrix W; ， The dynamically changing prior matrix is ​​scaled by To express: ， Among them, resize represents the bilinear interpolation method, i∈[2,5].

3. The method according to claim 2, characterized in that The jointly optimizing the features of the dynamically changing prior matrix and the output feature map and outputting the optimized output feature map comprises: Will The features of are concatenated in the channel dimension to obtain : ， Use a dynamically changing prior matrix to guide the spatial attention features of the feature maps at each stage : ， The feature pyramid structure is introduced into the feature extraction network, and each layer of the pyramid is represented as : ， Conv stands for convolution, and upsample stands for upsampling, which is used to amplify deep features to the same size as shallow features using bilinear interpolation.

4. The method according to claim 3, characterized in that The step of training the feature extraction network according to the optimized output feature map and the marked defect part includes: right Upsample and convolve to get the final output feature map , which represents the probability that each pixel belongs to the defect area: ， In obtaining Afterwards, Use the cross entropy loss function with the marked binary defect Y Perform calculations to supervise the training of the network: 。 5. The method according to claim 4, characterized in that The step of inputting the wafer image to be inspected into the trained feature extraction network and outputting the wafer defect inspection result comprises: After the network training is completed, the image to be detected is input into the trained network to obtain the detection result of the image, and the detection result is filtered using a threshold. The area greater than the threshold is the defect area, and the area less than the threshold is the background area.

6. A system for detecting wafer defects based on attention pyramid changes, using the method described in any one of claims 1 to 5, characterized in that: include: A data processing module, used to pre-process the pre-shot wafer image and mark the defective part of the pre-processed wafer image; A network construction module, used for constructing a feature extraction network, inputting the defect part into the feature extraction network, and obtaining an output feature map; A deep feature optimization module, used for optimizing the deep features in the output feature map; Dynamic prior module, used to generate a dynamically changing prior matrix based on the optimized deep features; A joint optimization module, used for performing joint optimization based on the characteristics of the dynamically changing prior matrix and the output feature map, and outputting an optimized output feature map; A training module, used for training the feature extraction network according to the optimized output feature map and the marked defect part; The detection module is used to input the wafer image to be detected into the trained feature extraction network and output the wafer defect detection result.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: The processor runs the computer program to implement the method according to any one of claims 1 to 4.

Images