Method and system for zero-shot wafer defect detection based on multi-scale feature difference comparison

Abstract

The application discloses a kind of multi-scale feature difference comparison's zero sample wafer defect detection method and system, method includes: obtaining preprocessed standard image and image to be examined, and standard image and image to be examined are aligned in space;Pre-trained deep convolutional neural network is constructed, the network weight of fixed deep convolutional neural network is extracted using deep convolutional neural network to standard image and image to be examined with multi-level feature, and respectively the feature map of each level is obtained;Based on feature map, the probability vector of the same space position of standard image and image to be examined is calculated respectively, and the divergence value is calculated according to probability vector, and difference map is generated;Different levels of difference map are fused, and comprehensive difference map is obtained, and the binary defect positioning map is output after processing comprehensive difference map, for automatic optical inspection.The application greatly reduces the dependence on data, simplifies deployment process, shortens deployment cycle and reduces application cost.

Description

Technical Field

[0001] This invention relates to the field of semiconductor defect detection technology, specifically to a zero-sample wafer defect detection method and system based on multi-scale feature difference comparison. Background Technology

[0002] In semiconductor wafer manufacturing, automated optical inspection (AOI) is a crucial step in ensuring product yield. It is used to quickly and non-destructively detect physical defects on the wafer surface, such as scratches, particle contamination, missing patterns, and bridging. Traditional AOI systems and many emerging deep learning solutions typically require collecting large amounts of data for specific production lines or patterns and training models, which is cumbersome and costly. Especially for high-yield production lines, defect samples are scarce and difficult to obtain, posing a significant challenge to supervised or weakly supervised learning methods.

[0003] Currently, before performing wafer defect detection, it is necessary to collect normal wafer samples, extract their depth features, and establish a normal feature template library. During detection, defects are identified by calculating the similarity between the features of the sample to be tested and multiple normal feature templates in the normal feature template library, combined with an adaptive threshold. While this method avoids the need for defective samples, it still heavily relies on the construction of a normal sample template library. Summary of the Invention

[0004] Based on this, the purpose of this invention is to provide a zero-sample wafer defect detection method and system for multi-scale feature difference comparison, which aims to solve the problems of scarce and difficult-to-obtain defect samples and heavy reliance on the construction of normal sample template libraries in current wafer defect detection methods.

[0005] To achieve the above objectives, this invention proposes a zero-sample wafer defect detection method based on multi-scale feature difference comparison. The zero-sample wafer defect detection method based on multi-scale feature difference comparison includes:

[0006] Obtain a preprocessed standard image and an image to be inspected, and spatially align the standard image and the image to be inspected.

[0007] Construct a pre-trained deep convolutional neural network, fix the network weights of the deep convolutional neural network, use the deep convolutional neural network to perform multi-level feature extraction on the standard image and the image to be inspected, and obtain the feature map of each level respectively.

[0008] Based on the feature map, probability vectors for the same spatial locations in the standard image and the image to be inspected are calculated respectively. Divergence values ​​are calculated based on the probability vectors, and a difference map is generated.

[0009] Fusing different levels of difference maps to obtain a comprehensive difference map, post-processing the comprehensive difference map, and outputting a binary defect positioning map for automatic optical inspection.

[0010] According to an aspect of the above technical solution, the step of constructing the pre-trained deep convolutional neural network comprises:

[0011] A pre-trained deep convolutional neural network is constructed, the deep convolutional neural network comprising an input layer and a plurality of consecutive residual stages, the input layer comprising a convolutional layer and a pooling layer, and any of the residual stages being stacked by a plurality of residual blocks, and any of the residual blocks at least containing a convolutional layer, a batch normalization layer, and an activation function.

[0012] According to an aspect of the above technical solution, the step of using the deep convolutional neural network to perform multi-level feature extraction on the standard image and the to-be-inspected image and obtaining a feature map of each level respectively comprises:

[0013] The standard image and the to-be-inspected image are input into the deep convolutional neural network, in a forward propagation cycle of the deep convolutional neural network, shallow residual stages, middle residual stages, and deep residual stages are selected from the plurality of residual stages, and feature maps output by the selected residual stages are captured in the selected residual stages.

[0014] According to an aspect of the above technical solution, the step of calculating a probability vector of the same spatial position of the standard image and the to-be-inspected image based on the feature map, calculating a divergence value based on the probability vector, and generating a difference map comprises:

[0015] Based on a standard feature map of the standard image and a to-be-inspected feature map of the to-be-inspected image output by the same residual stage, for each spatial position , a C-dimensional standard feature vector in the standard feature map and a C-dimensional to-be-inspected feature vector in the to-be-inspected feature map are extracted.

[0016] According to an aspect of the above technical solution, after the standard probability vector and the to-be-inspected probability vector are obtained, for each spatial position , a divergence value of the to-be-inspected probability vector mapped on the standard probability vector is calculated, and summation is performed on all channels:

[0017]

[0018] wherein, is a probability value of the standard probability vector distributed on the cth channel, is the natural logarithm of the probability value, is the natural logarithm of the probability value of the probability vector distribution of the to-be-inspected probability on the cth channel.

[0019] According to an aspect of the above technical solution, the step of generating the difference map comprises:

[0020] According to each spatial position in the standard feature map and the to-be-inspected feature map , the spatial position The divergence value calculated is filled in a matrix of a preset size based on the spatial coordinates of the spatial position, wherein the size of the matrix is consistent with the spatial size of the feature map;

[0021] Collecting different levels of difference maps obtained by the selected residual stage calculation, obtaining a group of multi-scale difference representations.

[0022] According to an aspect of the above technical solution, the step of fusing different levels of difference maps, obtaining a comprehensive difference map, post-processing the comprehensive difference map, and outputting a binary defect positioning map for automatic optical detection comprises:

[0023] The size of the several difference maps in the difference representation is restored to the same size as the to-be-inspected image, and the several difference maps of the restored size are fused into a comprehensive difference map;

[0024] An adaptive threshold method is used to binarize the comprehensive difference map, and an optimal threshold value of the comprehensive difference map is obtained, the defect area in the comprehensive difference map is marked according to the optimal threshold value, morphological operations are applied to the binarized comprehensive difference map, small noise points are removed and holes in the defect area are filled, and a binary defect positioning map is output for automatic optical detection.

[0025] The application also provides a multi-scale feature difference comparison zero-sample wafer defect detection system, which is used to implement the multi-scale feature difference comparison zero-sample wafer defect detection method.

[0026] The alignment module is used to obtain the preprocessed standard image and the to-be-inspected image, and aligns the standard image and the to-be-inspected image in space.

[0027] The extraction module is used to construct a pre-trained deep convolutional neural network, fix the network weight of the deep convolutional neural network, use the deep convolutional neural network to perform multi-level feature extraction on the standard image and the to-be-inspected image, and obtain a feature map of each level, respectively.

[0028] A computing module is configured to calculate a probability vector of the same spatial position of the standard image and the to-be-detected image based on the feature map, calculate a divergence value according to the probability vector, and generate a difference map;

[0029] An output module is configured to fuse the difference maps of different levels to obtain a comprehensive difference map, post-process the comprehensive difference map, and output a binary defect positioning map for automatic optical detection.

[0030] The application further provides a computer readable storage medium, which stores a computer program, and the program is executed by a processor to implement the multi-scale feature difference comparison based zero-sample wafer defect detection method.

[0031] The application further provides an electronic device, which comprises a memory, a processor and a computer program stored in the memory and running on the processor, and the processor implements the multi-scale feature difference comparison based zero-sample wafer defect detection method when executing the computer program.

[0032] In summary, according to the multi-scale feature difference comparison based zero-sample wafer defect detection method provided by the application, the training or modeling stage is completely abandoned, a fixed and pre-trained deep convolutional neural network on a general data set is used as a feature extractor, zero-sample anomaly detection and positioning are realized by directly calculating the difference between a to-be-detected image and a defect-free reference image in a multi-scale deep feature space, a plurality of feature maps of the pre-trained network are extracted, a difference value between corresponding feature maps is calculated layer by layer by using a metric function, a plurality of difference maps are generated, the plurality of difference maps of different resolutions are up-sampled to the same scale and fused by weighting to generate a comprehensive difference map with abnormality, and the comprehensive difference map is used for automatic optical detection. The pre-trained model with frozen weights is directly used as the feature extractor, the result can be obtained through one-time forward propagation and efficient feature difference calculation, the real plug-and-play is realized, only one or a few gold reference images are needed to work, the dependence on data is greatly reduced, the deployment process is simplified, the deployment cycle is shortened, and the application cost is reduced.

[0033] Additional aspects and advantages of the application will be made apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0034] Figure 1 A flowchart of the multi-scale feature difference comparison based zero-sample wafer defect detection method in the embodiment one of the application;

[0035] Figure 2 A structural schematic diagram of the multi-scale feature difference comparison based zero-sample wafer defect detection system in the embodiment two of the application;

[0036] Figure 3 Figure 1 is a block diagram of the structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0037] For the purpose of promoting an understanding of the principles of the application, reference will now be made to the embodiments illustrated in the drawings. It is expressly understood that the drawings are presented for the purpose of illustration and description only and are not intended as a definition of the limits of the application. As illustrated in the accompanying drawings, several embodiments of the application are shown. However, it should be apparent that the application can be embodied in a wide variety of forms not explicitly shown and that the specific embodiments disclosed are merely representative.

[0038] It is to be understood that the herein described components (for example elements of a product), can also be in the general form of a product of manufacture (for example a bar, a cartridge, a cassette, etc.) that contains, or stores the several components for the end use product. As such, when the component (or elements of a product) is referred to as being "positioned" "disposed" "connected" or "attached" to another component, it is understood in light of this disclosure that the positioning, disposition, connection or attachment can be directly to the other component or indirectly via an intervening component as would be recognized by those of ordinary skill in the art. It will be appreciated that the terms "on", "right", "left", "vertical", "horizontal", "top", "bottom", "under" and the like are used herein for ease of description to describe the aspects of an embodiment as illustrated in the drawings.

[0039] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in the description of the application herein is for describing particular embodiments only and is not intended to be limiting of the application. All publications, patent applications, patents, and other references mentioned herein are incorporated by reference in their entirety for the teachings relevant to the sentence and / or paragraph in which the reference is presented.

[0040] Embodiment One

[0041] Referring to Figure 1, a flow chart of a multi-scale feature difference comparison zero-sample wafer defect detection method according to an embodiment of the present application is shown. The multi-scale feature difference comparison zero-sample wafer defect detection method includes the following steps S01-S04: Figure 1 S01, obtaining a pre-processed standard image and a to-be-inspected image, and spatially aligning the standard image and the to-be-inspected image.

[0042] A known gold reference image is obtained as a standard image, and a to-be-inspected image is obtained. The standard image and the to-be-inspected image are pre-processed as necessary. The pre-processing methods include, but are not limited to, image size normalization, color space conversion (such as conversion to a grayscale image), and fine image registration (Image Registration), to ensure that the two images are completely aligned in space.

[0043]

[0044] ​S02, constructing a pre-trained deep convolutional neural network, fixing the network weights of the deep convolutional neural network, using the deep convolutional neural network to perform multi-level feature extraction on the standard image and the to-be-detected image, and respectively acquiring a feature map of each level.

[0045] In this embodiment, the deep convolutional neural network constructed needs to learn general and powerful feature representation in advance on millions of natural images, and the features have high semantic information and can effectively distinguish "normal" from "abnormal" and not specific types of defects, completely abandoning any form of training or offline modeling phase. Therefore, for a completely new defect type that has never appeared in the training set, as long as its feature representation is different from that of the normal area, the method can detect it, thereby having strong generalization ability.

[0046] It should be noted that, in order to preserve the pre-training knowledge and maintain the consistency of the extracted features, the network weights of the deep convolutional neural network in this embodiment are fixed and not fine-tuned based on target data. The general feature representation ability learned by the network on a large data set is used to avoid overfitting or knowledge forgetting caused by fine-tuning on a small-scale or specific target data set, and the deep convolutional neural network is used as a fixed feature extractor. Then, after inputting the standard image and the to-be-detected image, the features are extracted by forward propagation and compared to output the result, achieving the purpose of plug-and-play.

[0047] Preferably, the deep convolutional neural network ResNet-50 is taken as an example for illustration in this embodiment. The deep convolutional neural network includes an input layer and a plurality of continuous residual stages. The input layer includes a convolutional layer and a pooling layer. The convolutional kernel of the convolutional layer has a size of 7x7, a step of 2, and an output channel number of 64. The convolutional layer maps a single-channel grayscale image into a 64-channel feature map, captures basic edge and texture features, and at the same time, the step of 2 realizes the first down-sampling, reducing the input image size to 1 / 2. The pooling kernel of the pooling layer has a size of 3x3 and a step of 2. The pooling layer further down-samples the input image size to 1 / 2 again, enhances the translation invariance of the features, and filters local noise.

[0048] Each residual stage is composed of stacked residual blocks, and each residual block contains at least a convolutional layer, a batch normalization layer, and an activation function. In this embodiment, four consecutive residual stages are used: layer1, layer2, layer3, and layer4. Layer1 contains 3 residual blocks, with 256 output feature map channels and a large spatial size, preserving rich detail information. Layer2 contains 4 residual blocks, with 512 output channels and a halved spatial size. Layer3 contains 6 residual blocks, with 1024 output channels and a halved spatial size again, capturing mid-level semantic features. Layer4 contains 3 residual blocks, with 2048 output channels and the smallest spatial size, containing highly abstract high-level semantic features.

[0049] In this step, since the deep convolutional neural network only utilizes the feature maps of the intermediate layers, the output layer of the deep convolutional neural network is discarded. Simultaneously, a multi-output model is created to achieve parallel multi-level feature extraction, inputting the image into this reconstruction model all at once. During one forward propagation cycle of the network, the input data sequentially passes through all layers, capturing the output tensors at multiple specified intermediate layer locations; these tensors are the required multi-scale feature maps. To achieve multi-scale coverage from details to intermediate semantics to deep semantics, ensuring that wafer defects of different types and sizes can be captured, this embodiment selects several residual stages: shallow residual stage, intermediate residual stage, and deep residual stage, namely shallow residual stage layer1, intermediate residual stage layer3, and deep residual stage layer4, and captures the feature maps output by these residual stages.

[0050] S03. Based on the feature map, calculate the probability vectors of the same spatial locations in the standard image and the image to be inspected, calculate the divergence value based on the probability vectors, and generate a difference map.

[0051] For the standard feature map of the standard image and the feature map to be detected of the image output in the same residual stage in step S02, calculate the difference between the standard feature map and the feature map to be detected in different residual stages. The calculation method includes, but is not limited to, the Kullback-Leibler divergence used in this embodiment. Other metrics such as Jensen-Shannon divergence, Euclidean distance (L2Distance), cosine distance (Cosine Distance), or correlation coefficient can also be used. The specific calculation process is as follows:

[0052] Based on the standard feature map of the standard image output from the same residual stage and the feature map of the image to be inspected, for each spatial location... Extract the C-dimensional standard feature vector from the standard feature map. , and the C-dimensional feature vector in the feature map to be detected , the standard feature vector and the feature vector to be detected are converted into standard probability vectors and to-be-detected probability vectors respectively using a conversion function The conversion function used in the embodiment is Softmax:

[0053]

[0054]

[0055] The calculation formula of the Softmax function is as follows: , so that the sum of all elements is 1.

[0056] After obtaining the standard probability vector and the to-be-detected probability vector, for each spatial position , the divergence value of the to-be-detected probability vector mapped on the standard probability vector is calculated, and the sum of all channels is calculated:

[0057]

[0058] , wherein is the probability value of the standard probability vector distributed on the cth channel, is the natural logarithm of the probability value, is the natural logarithm of the probability value of the to-be-detected probability vector distributed on the cth channel.

[0059] In actual calculation, in order to prevent numerical error caused by taking logarithm of zero , a small positive number epsilon (ε, for example, 1e-8) is clamped to the probability value before calculating the logarithm, that is: The actual calculation of is .

[0060] Each spatial position in the feature map is traversed , and the corresponding KL divergence value is calculated according to the above formula . The calculation results of all positions are filled into a two-dimensional matrix according to the original spatial coordinates, and finally a difference map completely consistent with the spatial size of the layer feature map is generated. Each pixel value on the map directly quantifies the feature dissimilarity of the input image pair in the corresponding local area. The generation process of the difference map can be uniformly represented by the following mathematical transformation:

[0061]

[0062] ​

[0063]

[0064] Collecting different levels of difference maps calculated by selected residual stages, obtaining a set of multi-scale difference representations.

[0065] S04, fusing different levels of difference maps to obtain a comprehensive difference map, post-processing the comprehensive difference map, and outputting a binary defect positioning map for automatic optical inspection.

[0066] Since the difference maps at each level in step S03 have different spatial resolutions, the low-resolution difference maps need to be upsampled to restore the size of the difference representations to the same size as the image to be inspected, and the restored size of the difference maps is fused into a comprehensive difference map by weighted summation, wherein the weight coefficients in the weighted summation process can be artificially assigned according to the importance of different levels of features or determined by a simple optimization strategy. In addition, the fusion method in this step can also use a multi-scale image fusion algorithm (such as Laplacian pyramid fusion) or use a simple small convolutional neural network for adaptive fusion.

[0067] The obtained comprehensive difference map is post-processed to generate a final binary defect positioning map. Specifically, an adaptive threshold method is used to binarize the comprehensive difference map to obtain an optimal threshold of the comprehensive difference map, or a fixed threshold is set to binarize the difference map. Based on the optimal threshold or the fixed threshold, the region with a pixel value greater than the optimal threshold or the fixed threshold is determined as a potential defect region. Morphological operations are applied to the binarized comprehensive difference map, which can use opening and closing operations, etc. The opening operation can remove small noise points and fill the holes in the defect region, and the closing operation can fill the holes in the defect region and smooth its contour, finally obtaining a connected and denoised defect binary mask.

[0068] Output the binary defect positioning map, where the white area (high pixel value) is identified as the detected abnormal position, and the black area (low pixel value) is the normal background. The system can be integrated into an AOI device to automatically trigger an alarm, record or sorting mechanism, and complete the automatic detection process.

[0069] In summary, according to the multi-scale feature difference comparison zero sample wafer defect detection method provided by the application, by completely abandoning the training or modeling stage, a fixed and pre-trained deep convolutional neural network on a general data set is used as a feature extractor, zero sample anomaly detection and positioning are realized by directly calculating the difference between the to-be-detected image and the defect-free reference image in the multi-scale deep feature space, the feature maps of multiple levels of the pre-trained network are extracted, a metric function is used to calculate the difference value between the corresponding feature maps layer by layer, and multiple difference maps are generated, the multiple difference maps of different resolutions are up-sampled to the same scale and weighted and fused to generate a comprehensive difference map with obvious abnormalities, which is used for automatic optical detection. The application directly uses a general pre-trained model with frozen weights as a feature extractor, and the result can be obtained through one forward propagation and efficient feature difference calculation, realizing true plug-and-play, and only one to several gold reference images are needed to work, greatly reducing the dependence on data, simplifying the deployment process, shortening the deployment cycle and reducing the application cost.

[0070] Embodiment two

[0071] Another aspect of the application also provides a multi-scale feature difference comparison zero sample wafer defect detection system, please refer to Figure 2 , which is a structural schematic diagram of the multi-scale feature difference comparison zero sample wafer defect detection system in the second embodiment of the application, the multi-scale feature difference comparison zero sample wafer defect detection system comprises:

[0072] The alignment module 11 is used to acquire the preprocessed standard image and the to-be-detected image, and align the standard image and the to-be-detected image in space.

[0073] The extraction module 12 is used to construct a pre-trained deep convolutional neural network, fix the network weight of the deep convolutional neural network, perform multi-level feature extraction on the standard image and the to-be-detected image using the deep convolutional neural network, and acquire the feature map of each level respectively.

[0074] The calculation module 13 is used to calculate the probability vector of the same spatial position of the standard image and the to-be-detected image based on the feature map respectively, calculate the divergence value according to the probability vector, and generate a difference map.

[0075] The output module 14 is used to fuse the difference maps of different levels, acquire a comprehensive difference map, post-process the comprehensive difference map, and output a binary defect positioning map for automatic optical detection.

[0076] A known golden reference image is acquired as a standard image, and a to-be-inspected image is acquired, and the standard image and the to-be-inspected image are preprocessed as required, the preprocessing manner including but not limited to: image size normalization, color space conversion (such as conversion into a gray image), and fine image registration (Image Registration), so as to ensure that the two images are completely aligned in space.

[0077] In the embodiment, the constructed deep convolutional neural network needs to learn a general and powerful feature representation in advance on millions of natural images, the features have high semantic information and can effectively distinguish "normal" from "abnormal", and any form of training or offline modeling phase is completely abandoned. Therefore, for a completely new defect type that has never appeared in the training set, as long as the feature representation of the defect type is different from that of the normal area, the method can detect the defect type, thereby having strong generalization ability.

[0078] It should be noted that, in order to retain the pre-training knowledge and maintain the consistency of the extracted features, the network weight of the deep convolutional neural network in the embodiment is fixed and is not fine-tuned based on target data, the general feature representation ability learned by the network on a large data set is used, overfitting or knowledge forgetting caused by fine-tuning on a small-scale or specific target data set is avoided, the deep convolutional neural network is used as a fixed feature extractor, and then the standard image and the to-be-inspected image are input, features are extracted and compared through forward propagation, and finally the result is output, thereby achieving the purpose of plug-and-play.

[0079] Preferably, the embodiment is described by taking the deep convolutional neural network ResNet-50 as an example, the deep convolutional neural network includes an input layer and a plurality of continuous residual stages, the input layer includes a convolution layer and a pooling layer, the size of the convolution kernel of the convolution layer is 7x7, the step is 2, the number of output channels is 64, the convolution layer maps a single-channel gray image into a 64-channel feature map, captures basic edge and texture features, and at the same time, the step 2 realizes first down-sampling, and the input image size is reduced to 1 / 2; the size of the pooling kernel of the pooling layer is 3x3, and the step is also 2, the pooling layer further down-samples, and the input image size is reduced to 1 / 2 again, the translation invariance of the features is enhanced, and local noise is filtered.

[0080] Any of the residual stages is stacked by several residual blocks, and any of the residual blocks at least comprises a convolution layer, a batch normalization layer, and an activation function. In the embodiment, four continuous residual stages are adopted, which are layer1, layer2, layer3, and layer4. The layer1 comprises 3 residual blocks, the output feature map channel number is 256, the spatial size is large, and rich detail information is reserved. The layer2 comprises 4 residual blocks, the output channel number is 512, and the spatial size is halved. The layer3 comprises 6 residual blocks, the output channel number is 1024, and the spatial size is halved again, so that the middle-level semantic features are captured. The layer4 comprises 3 residual blocks, the output channel number is 2048, and the spatial size is the smallest, so that the high-level semantic features are included.

[0081] In the step, the output layer of the deep convolutional neural network is discarded because the deep convolutional neural network only uses the feature maps of the intermediate layers. Meanwhile, a multi-output model is created to realize parallel multi-level feature extraction, and the input image is input into the reconstruction model at one time. In one forward propagation cycle of the network, the input data passes through all levels in sequence, and the output tensors at the specified multiple intermediate layer positions are captured, which are the required multi-scale feature maps. In order to realize the multi-scale coverage capability of details→middle-level semantics→deep-level semantics, and to ensure that different types and sizes of wafer defects can be captured, the embodiment selects several shallow residual stages, middle residual stages, and deep residual stages in the residual stages, that is, the shallow residual stage layer1, the middle residual stage layer3, and the deep residual stage layer4, and captures the feature maps output by the selected residual stages.

[0082] For the standard feature maps of the standard image and the detection feature maps of the detection image output by the same residual stage in step S02, the differences between the standard feature maps and the detection feature maps in different residual stages are calculated. The calculation methods include but are not limited to the Kullback-Leibler divergence adopted in the embodiment, and other measurement methods such as Jensen-Shannon divergence, Euclidean distance (L2 Distance), cosine distance (Cosine Distance), or correlation coefficient can also be used. The specific calculation process is as follows:

[0083] Based on the standard feature maps of the standard image and the detection feature maps of the detection image output by the same residual stage, for each spatial position , the C-dimensional standard feature vector in the standard feature map and the C-dimensional detection feature vector in the detection feature map are extracted, and the standard feature vector and the detection feature vector are converted into standard probability vectors and the to-be-tested probability vector The conversion function used in the embodiment is Softmax:

[0084]

[0085]

[0086] The calculation formula of the Softmax function is: , which ensures that the sum of all elements is 1.

[0087] After obtaining the standard probability vector and the to-be-tested probability vector, for each spatial position , the divergence value of the to-be-tested probability vector mapped on the standard probability vector is calculated, and the sum of all channels is calculated:

[0088]

[0089] wherein, is the probability value of the standard probability vector distributed on the cth channel, is the natural logarithm of the probability value, is the natural logarithm of the probability value of the to-be-tested probability vector distributed on the cth channel.

[0090] In actual calculation, in order to prevent numerical errors caused by taking the logarithm of zero , a small positive number epsilon (ε, for example, 1e-8) is clamped to the probability value before calculating the logarithm, that is: The actual calculation of .

[0091] Each spatial position in the feature map is traversed , and the corresponding KL divergence value is calculated according to the above formula . The calculation results of all positions are filled into a two-dimensional matrix according to the original spatial coordinates, and finally a difference map completely consistent with the spatial size of the layer feature map is generated. Each pixel value on the map directly quantifies the feature dissimilarity of the input image pair in the corresponding local area. The generation process of the difference map can be uniformly represented by the following mathematical transformation:

[0092]

[0093]

[0094]

[0095] Collect different levels of difference maps calculated by selected residual stages to obtain a group of multi-scale difference representations.

[0096] Since the difference maps of different levels in step S03 have different spatial resolutions, the low-resolution difference maps need to be up-sampled to restore the size of the difference maps to the same size as the image to be detected, and the difference maps of the restored size are fused into a comprehensive difference map by weighted summation, wherein the weight coefficients in the weighted summation process can be artificially assigned according to the importance of different features or determined by a simple optimization strategy. In addition, the fusion method in this step can also use a multi-scale image fusion algorithm (such as Laplacian pyramid fusion) or an adaptive fusion using a simple small convolutional neural network.

[0097] The obtained comprehensive difference map is post-processed to generate a final binary defect positioning map. Specifically, an adaptive threshold method is used to binarize the comprehensive difference map to obtain an optimal threshold of the comprehensive difference map, or a fixed threshold is set to binarize the difference map. Based on the optimal threshold or the fixed threshold, the region with a pixel value greater than the optimal threshold or the fixed threshold is determined as a potential defect region. Morphological operations are applied to the binarized comprehensive difference map, and operations such as opening and closing operations can be used. The opening operation can remove small noise points and fill the holes in the defect region, and the closing operation can fill the holes in the defect region and smooth the contour, finally obtaining a connected and denoised defect binary mask.

[0098] The binary defect positioning map is output, wherein the white region (high pixel value) is identified as the detected abnormal position, and the black region (low pixel value) is the normal background. The system can be integrated into an AOI device to automatically trigger an alarm, recording or sorting mechanism, and complete the automatic detection process.

[0099] In summary, according to the zero-sample wafer defect detection system based on multi-scale feature difference comparison provided by the present application, by completely abandoning the training or modeling stage, a fixed and pre-trained deep convolutional neural network on a general data set is used as a feature extractor, and by directly calculating the difference between the image to be detected and the defect-free reference image in the multi-scale deep feature space, zero-sample anomaly detection and positioning are realized. The features of multiple levels of the pre-trained network are extracted, and a metric function is used to calculate the difference value between the corresponding feature maps layer by layer to generate multiple difference maps. The multiple difference maps of different resolutions are up-sampled to the same size and fused by weighted summation to generate a comprehensive difference map that highlights the abnormalities, which is used for automatic optical detection. The present application directly uses a general pre-trained model with frozen weights as a feature extractor, and the result can be obtained by one forward propagation and efficient feature difference calculation, realizing true plug-and-play. At the same time, only one to several golden reference images are needed to work, greatly reducing the dependence on data, simplifying the deployment process, shortening the deployment period and reducing the application cost.

[0100] Embodiment Three

[0101] Another aspect of the present application also provides a computer readable storage medium having stored thereon one or more computer programs which, when executed by a processor, implement the above-mentioned zero-sample wafer defect detection method based on multi-scale feature difference comparison.

[0102] Those skilled in the art can understand that the logic or steps represented in the flowchart or otherwise described herein, for example, can be considered as a list of executable instructions for implementing the logic function, which can be embodied in any computer readable storage medium for use by or in connection with an instruction execution system, apparatus or device, such as a computer-based system, a system including a processor or other system that can fetch the instructions from the instruction execution system, apparatus or device and execute the instructions, or in conjunction with these instruction execution systems, apparatus or devices. For the present specification, the "computer readable storage medium" can be any device that can contain, store, communicate, propagate or transport programs for use by or in connection with an instruction execution system, apparatus or device, or in conjunction with these instruction execution systems, apparatus or devices.

[0103] More specific examples (a non-exhaustive list) of the computer readable storage medium include the following: an electrical connection having one or more wires (electrical devices), a portable computer diskette (magnetic devices), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). In addition, the computer readable storage medium can even be paper or other suitable medium on which the program can be printed, because the program can be electronically obtained, for example, by optical scanning of the paper or other medium, followed by electronic conversion, interpretation or processing, if necessary, in other suitable manner, and then stored in a computer memory.

[0104] Embodiment Four

[0105] Figure 3 A structural block diagram of an electronic device provided for embodiment four. The electronic device includes a memory, a processor, and a computer program stored on the memory and executable on the processor, and the processor executes the program to implement the above-mentioned zero-sample wafer defect detection method based on multi-scale feature difference comparison in the embodiments. Figure 3 The electronic device 30 shown is only an example and should not bring any limitation to the function and use range of the embodiments of the present application.

[0106] As Figure 3As shown, the electronic device 30 can be in the form of a general computing device, for example, it can be a server device. The components of the electronic device 30 can include, but are not limited to, the at least one processor 31 described above, the at least one memory 32 described above, a bus 33 that connects the different system components, including the memory 32 and the processor 31.

[0107] The bus 33 includes a data bus, an address bus, and a control bus.

[0108] The memory 32 can include volatile memory, such as RAM 321 (Random Access Memory), and / or cache memory 322, and can further include non-volatile memory, such as ROM 323 (Read Only Memory).

[0109] The memory 32 can also include a program tool 325 having a set of (at least one) program modules 324, including but not limited to: an operating system, one or more application programs, other program modules, and program data, and each of these examples, or some combination thereof, can include implementation of a network environment.

[0110] The processor 31 performs various function applications and data processing by running the computer program stored in the memory 32, such as the multi-scale feature difference comparison zero-sample wafer defect detection method of the present application as described above.

[0111] The electronic device 30 can also communicate with one or more external devices 34 (such as a keyboard, a pointing device, etc.) via an I / O interface 35 (Input / Output interface). And, the electronic device 30 can also communicate with one or more networks (such as a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 36. As Figure 3 As shown, the network adapter 36 communicates with other modules of the electronic device 30 for model generation via the bus 33. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with the electronic device 30 for model generation, including but not limited to: microcode, device drivers, redundant processors, disk drive arrays, RAID (Redundant Array of Independent Disks) systems, tape drives, and data backup storage systems, etc.

[0112] It should be noted that although several units / modules or sub-units / modules of the electronic device are mentioned in the above detailed description, such division is only exemplary and not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more units / modules described above can be embodied in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided into multiple units / modules.

[0113] In the description of the present specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0114] The above-described embodiments only express several implementation manners of the present application, which are described in a more specific and detailed manner, but cannot be understood as a limitation on the patent scope of the present application. It should be noted that, for those of ordinary skill in the art, several modifications and improvements can be made without departing from the concept of the present application, which are all within the protection scope of the present application. Therefore, the protection scope of the patent of the present application should be subject to the appended claims.

Claims

1. A multi-scale feature difference comparison based zero-shot wafer defect detection method, characterized in that, The zero-sample wafer defect detection method based on multi-scale feature difference comparison comprises the following steps: acquiring a preprocessed standard image and a to-be-detected image, and spatially aligning the standard image and the to-be-detected image; constructing a pre-trained deep convolutional neural network, fixing the network weight of the deep convolutional neural network, using the deep convolutional neural network to perform multi-level feature extraction on the standard image and the to-be-detected image, and acquiring a feature map of each level respectively; Based on the feature map, the probability vectors of the same spatial position of the standard image and the to-be-detected image are calculated respectively, the divergence value is calculated according to the probability vectors, and a difference map is generated, based on the standard feature map of the standard image and the to-be-detected feature map of the to-be-detected image output by the same residual stage, for each spatial position (i, j), a C-dimensional standard feature vector vref(i, j) in the standard feature map and a C-dimensional to-be-detected feature vector vtest(i, j) in the to-be-detected feature map are extracted, the standard feature vector and the to-be-detected feature vector are converted into a standard probability vector and a to-be-detected probability vector respectively using a conversion function, after obtaining the standard probability vector and the to-be-detected probability vector, for each spatial position , the divergence value of the to-be-detected probability vector mapped on the standard probability vector is calculated, and summation is performed for all channels: wherein, is the probability value of the standard probability vector distribution on the cth channel, is the natural logarithm of the probability value, is the natural logarithm of the probability value of the to-be-detected probability vector distribution on the cth channel, according to each spatial position in the standard feature map and the to-be-detected feature map , the spatial position , the divergence value calculated by the spatial position, filled in a matrix of a preset size based on the spatial coordinates of the spatial position, wherein the size of the matrix is consistent with the spatial size of the feature map, and each spatial position in the feature map is traversed , the calculation results of all positions are filled into a two-dimensional matrix according to the original spatial coordinates, to generate a difference map completely consistent with the spatial size of the feature map of the layer, the difference map , the generation process can be uniformly represented by the following mathematical transformation: collecting difference maps of different levels calculated by selected residual stages, and acquiring a group of multi-scale difference representations; fusing the difference maps of different levels to acquire a comprehensive difference map, post-processing the comprehensive difference map, and outputting a binary defect positioning map for automatic optical detection.

2. The method of claim 1, wherein the multi-scale feature difference comparison is performed by a convolutional neural network (CNN) trained on a dataset of labeled training images. The step of constructing the pre-trained deep convolutional neural network comprises the following steps: constructing a pre-trained deep convolutional neural network, wherein the deep convolutional neural network comprises an input layer and a plurality of continuous residual stages, the input layer comprises a convolutional layer and a pooling layer, any residual stage is stacked by a plurality of residual blocks, and any residual block at least contains a convolutional layer, a batch normalization layer and an activation function.

3. The method of claim 2, wherein the multi-scale feature difference comparison is performed by a convolutional neural network (CNN). 3 The step of using the deep convolutional neural network to perform multi-level feature extraction on the standard image and the to-be-detected image and acquiring a feature map of each level respectively comprises the following steps: inputting the standard image and the to-be-detected image into the deep convolutional neural network, selecting shallow residual stages, middle residual stages and deep residual stages in the plurality of residual stages in one forward propagation period of the deep convolutional neural network, and capturing feature maps output by the selected residual stages in the selected residual stages.

4. The zero-sample wafer defect detection method based on multi-scale feature difference comparison according to claim 1, characterized in that, The step of fusing the difference maps of different levels to acquire a comprehensive difference map, post-processing the comprehensive difference map, and outputting a binary defect positioning map for automatic optical detection comprises the following steps: restoring the size of a plurality of difference maps in the difference representation to the same size as the to-be-detected image, and fusing the plurality of difference maps with restored size into a comprehensive difference map; applying an adaptive threshold method to the comprehensive difference map to binarize the comprehensive difference map, acquiring an optimal threshold value of the comprehensive difference map, marking a defect area in the comprehensive difference map according to the optimal threshold value, applying a morphological operation to the binarized comprehensive difference map to remove tiny noise points and fill holes in the defect area, and outputting a binary defect positioning map for automatic optical detection.

5. A multi-scale feature difference-based zero-shot wafer defect detection system, comprising: The zero-sample wafer defect detection system based on multi-scale feature difference comparison is used to implement the zero-sample wafer defect detection method based on multi-scale feature difference comparison according to any one of claims 1-4, and the system comprises: an alignment module configured to acquire a preprocessed standard image and a to-be-detected image, and spatially align the standard image and the to-be-detected image; an extraction module configured to construct a pre-trained deep convolutional neural network, fix the network weight of the deep convolutional neural network, use the deep convolutional neural network to perform multi-level feature extraction on the standard image and the to-be-detected image, and acquire a feature map of each level respectively; The computing module is configured to calculate a probability vector of the same spatial position of the standard image and the image to be detected based on the feature map, calculate a divergence value according to the probability vector, and generate a difference map, extract a C-dimensional standard feature vector vref(i, j) in the standard feature map and a C-dimensional test feature vector vtest(i, j) in the test feature map for each spatial position (i, j) based on the standard feature map of the standard image and the test feature map of the image to be detected output by the same residual stage, convert the standard feature vector and the test feature vector into a standard probability vector and a test probability vector respectively using a conversion function, and obtain the standard probability vector and the test probability vector , calculate a divergence value of the test probability vector mapped on the standard probability vector, and sum all channels. wherein, is the probability value of the standard probability vector distribution on the cth channel, is the natural logarithm of the probability value, is the natural logarithm of the probability value of the to-be-detected probability vector distribution on the cth channel, according to each spatial position in the standard feature map and the to-be-detected feature map , the spatial position , the divergence value calculated by the spatial position, is filled in a matrix of a preset size based on the spatial coordinates of the spatial position, wherein the size of the matrix is consistent with the spatial size of the feature map, and each spatial position in the feature map is traversed , the calculation results of all positions are filled into a two-dimensional matrix according to the original spatial coordinates, to generate a difference map completely consistent with the spatial size of the feature map of the layer, the difference map , in the generation process of the difference map, can be uniformly represented by the following mathematical transformation: collecting difference maps of different levels calculated by selected residual stages, and acquiring a group of multi-scale difference representations; An output module is configured to fuse the difference maps of different levels to obtain a comprehensive difference map, to post-process the comprehensive difference map, and to output a binary defect positioning map for automatic optical inspection.

6. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program, when executed by the processor, implements the zero-sample wafer defect detection method of multi-scale feature difference comparison according to any one of claims 1-4.

7. An electronic device comprising a memory, a processor, and a computer program stored on the memory and running on the processor, characterized in that, The processor, when executing the computer program, implements the zero-sample wafer defect detection method of multi-scale feature difference comparison according to any one of claims 1-4.

Images