Zero sample wafer defect detection method and system based on multi-scale characteristic difference comparison

Abstract

The invention discloses a zero sample wafer defect detection method and system based on multi-scale feature difference comparison, and the method comprises the steps: obtaining a preprocessed standard image and a to-be-detected image, and carrying out the spatial alignment of the standard image and the to-be-detected image; constructing a pre-trained deep convolutional neural network, fixing the network weight of the deep convolutional neural network, performing multi-level feature extraction on the standard image and the to-be-detected image by using the deep convolutional neural network, and respectively obtaining a feature map of each level; respectively calculating probability vectors of the same spatial positions of the standard image and the to-be-detected image based on the feature map, calculating divergence values according to the probability vectors, and generating a difference map; and fusing the difference images of different levels to obtain a comprehensive difference image, performing post-processing on the comprehensive difference image, and outputting a binary defect positioning image for automatic optical detection. According to the method, the dependence on data is greatly reduced, the deployment process is simplified, the deployment period is shortened, and the application cost is reduced.

Description

Technical Field

[0001] This invention relates to the field of semiconductor defect detection technology, specifically to a zero-sample wafer defect detection method and system based on multi-scale feature difference comparison. Background Technology

[0002] In semiconductor wafer manufacturing, automated optical inspection (AOI) is a crucial step in ensuring product yield. It is used to quickly and non-destructively detect physical defects on the wafer surface, such as scratches, particle contamination, missing patterns, and bridging. Traditional AOI systems and many emerging deep learning solutions typically require collecting large amounts of data for specific production lines or patterns and training models, which is cumbersome and costly. Especially for high-yield production lines, defect samples are scarce and difficult to obtain, posing a significant challenge to supervised or weakly supervised learning methods.

[0003] Currently, before performing wafer defect detection, it is necessary to collect normal wafer samples, extract their depth features, and establish a normal feature template library. During detection, defects are identified by calculating the similarity between the features of the sample to be tested and multiple normal feature templates in the normal feature template library, combined with an adaptive threshold. While this method avoids the need for defective samples, it still heavily relies on the construction of a normal sample template library. Summary of the Invention

[0004] Based on this, the purpose of this invention is to provide a zero-sample wafer defect detection method and system for multi-scale feature difference comparison, which aims to solve the problems of scarce and difficult-to-obtain defect samples and heavy reliance on the construction of normal sample template libraries in current wafer defect detection methods.

[0005] To achieve the above objectives, this invention proposes a zero-sample wafer defect detection method based on multi-scale feature difference comparison. The zero-sample wafer defect detection method based on multi-scale feature difference comparison includes: Obtain a preprocessed standard image and an image to be inspected, and spatially align the standard image and the image to be inspected. Construct a pre-trained deep convolutional neural network, fix the network weights of the deep convolutional neural network, use the deep convolutional neural network to perform multi-level feature extraction on the standard image and the image to be inspected, and obtain the feature map of each level respectively. Based on the feature map, probability vectors for the same spatial locations in the standard image and the image to be inspected are calculated respectively. Divergence values ​​are calculated based on the probability vectors, and a difference map is generated. By fusing difference maps at different levels, a comprehensive difference map is obtained. The comprehensive difference map is then post-processed, and a binarized defect location map is output for automatic optical inspection.

[0006] According to one aspect of the above technical solution, the step of constructing a pre-trained deep convolutional neural network includes: Construct a pre-trained deep convolutional neural network, the deep convolutional neural network including an input layer and several consecutive residual stages, the input layer including convolutional layers and pooling layers, any residual stage being composed of several stacked residual blocks, any residual block including at least a convolutional layer, a batch normalization layer, and an activation function.

[0007] According to one aspect of the above technical solution, the step of using the deep convolutional neural network to perform multi-level feature extraction on the standard image and the image to be inspected, and obtaining feature maps at each level respectively, includes: The standard image and the image to be inspected are input into the deep convolutional neural network. Within one forward propagation cycle of the deep convolutional neural network, a shallow residual stage, a middle residual stage, and a deep residual stage are selected from several residual stages, and the feature map output by the selected residual stage is captured in the selected residual stage.

[0008] According to one aspect of the above technical solution, the step of calculating probability vectors for the same spatial locations in the standard image and the image to be detected based on the feature map, calculating divergence values ​​based on the probability vectors, and generating a difference map includes: Based on the standard feature map of the standard image output from the same residual stage and the feature map of the image to be inspected, for each spatial location... Extract the C-dimensional standard feature vector from the standard feature map. and the C-dimensional feature vector in the feature map to be detected. The standard feature vector and the feature vector to be detected are converted into a standard probability vector and a probability vector to be detected, respectively, using a conversion function.

[0009] According to one aspect of the above technical solution, after obtaining the standard probability vector and the probability vector to be detected, for each spatial location... Calculate the divergence value of the probability vector to be detected mapped to the standard probability vector, and sum it over all channels:

[0010] in, It is the probability value of the standard probability vector distribution on the c-th channel. It is the natural logarithm of the probability value. It is the natural logarithm of the probability values ​​of the probability vector to be detected distributed on the c-th channel.

[0011] According to one aspect of the above technical solution, the step of generating the difference map includes: Based on each spatial location in the standard feature map and the feature map to be detected Spatial location The calculated divergence value is filled into a matrix of a preset size based on the spatial coordinates of the spatial location, wherein the size of the matrix is ​​consistent with the spatial size of the feature map; Collect difference maps at different levels obtained from the selected residual stages to obtain a set of multi-scale difference representations.

[0012] According to one aspect of the above technical solution, the steps of fusing difference maps at different levels to obtain a comprehensive difference map, post-processing the comprehensive difference map, and outputting a binarized defect location map for automatic optical inspection include: The dimensions of several difference maps within the difference representation are restored to the same size as the image to be inspected, and the restored difference maps are merged into a comprehensive difference map. An adaptive thresholding method is used to binarize the comprehensive difference map and obtain the optimal threshold for the comprehensive difference map. The defect areas in the comprehensive difference map are marked according to the optimal threshold. Morphological operations are applied to the binarized comprehensive difference map to remove small noise points and fill the holes in the defect areas, and a binarized defect location map is output for automatic optical inspection.

[0013] This invention also proposes a zero-sample wafer defect detection system based on multi-scale feature difference comparison. This system is used to implement the aforementioned zero-sample wafer defect detection method based on multi-scale feature difference comparison. The system includes: An alignment module is used to acquire a preprocessed standard image and an image to be inspected, and to spatially align the standard image and the image to be inspected. The extraction module is used to construct a pre-trained deep convolutional neural network, fix the network weights of the deep convolutional neural network, use the deep convolutional neural network to perform multi-level feature extraction on the standard image and the image to be inspected, and obtain the feature map of each level respectively. The calculation module is used to calculate the probability vectors of the same spatial locations in the standard image and the image to be detected based on the feature map, calculate the divergence value based on the probability vectors, and generate a difference map. The output module is used to fuse difference maps at different levels to obtain a comprehensive difference map, perform post-processing on the comprehensive difference map, and output a binarized defect location map for automatic optical inspection.

[0014] The present invention also proposes a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the zero-sample wafer defect detection method for multi-scale feature difference comparison as described above.

[0015] The present invention also proposes an electronic device, including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to implement the zero-sample wafer defect detection method for multi-scale feature difference comparison as described above.

[0016] In summary, the zero-shot wafer defect detection method based on multi-scale feature difference comparison provided by this invention completely eliminates the training or modeling stage. It employs a fixed, pre-trained deep convolutional neural network on a general dataset as the feature extractor. Zero-shot anomaly detection and localization are achieved by directly calculating the differences between the test image and a defect-free reference image in a multi-scale deep feature space. Feature maps from multiple layers of the pre-trained network are extracted, and a metric function is used to calculate the difference values ​​between corresponding feature maps layer by layer, generating multiple difference maps. These multiple difference maps at different resolutions are upsampled to the same scale and weighted and fused to generate a comprehensive difference map with significant anomalies for automatic optical inspection. This invention directly uses a general, frozen-weight pre-trained model as the feature extractor, obtaining results through a single forward propagation and efficient feature difference calculation, achieving true plug-and-play functionality. Furthermore, it only requires one or a few golden reference images to operate, greatly reducing data dependence, simplifying the deployment process, shortening the deployment cycle, and lowering application costs.

[0017] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0018] Figure 1 This is a flowchart of the zero-sample wafer defect detection method for multi-scale feature difference comparison in Embodiment 1 of the present invention; Figure 2 This is a schematic diagram of the zero-sample wafer defect detection system for multi-scale feature difference comparison in Embodiment 2 of the present invention; Figure 3 This is a structural block diagram of the electronic device in Embodiment 4 of the present invention. Detailed Implementation

[0019] To facilitate understanding of the present invention, a more complete description will be given below with reference to the accompanying drawings. Several embodiments of the invention are illustrated in the drawings. However, the invention can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete.

[0020] It should be noted that when a component is said to be "fixed to" another component, it can be directly on the other component or there may be an intervening component. When a component is said to be "connected to" another component, it can be directly connected to the other component or there may be an intervening component. The terms "vertical," "horizontal," "left," "right," and similar expressions used in this document are for illustrative purposes only.

[0021] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the specification of this invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. The term "and / or" as used herein includes any and all combinations of one or more of the associated listed items.

[0022] Example 1 Please see Figure 1 The diagram shows a flowchart of a zero-sample wafer defect detection method based on multi-scale feature difference comparison according to Embodiment 1 of the present invention. The zero-sample wafer defect detection method based on multi-scale feature difference comparison includes the following steps S01-S04, wherein: S01. Obtain the preprocessed standard image and the image to be inspected, and align the standard image and the image to be inspected in space.

[0023] Obtain a known gold reference image as the standard image, and an image to be inspected. Perform necessary preprocessing on the standard image and the image to be inspected. This preprocessing includes, but is not limited to, image size normalization, color space conversion (such as converting to grayscale), and fine image registration to ensure that the two images are perfectly aligned in space.

[0024] S02. Construct a pre-trained deep convolutional neural network, fix the network weights of the deep convolutional neural network, use the deep convolutional neural network to perform multi-level feature extraction on the standard image and the image to be inspected, and obtain the feature map of each level respectively.

[0025] In this embodiment, the constructed deep convolutional neural network needs to learn general and powerful feature representations in advance on millions of natural images. These features have high semantic information and can effectively distinguish between "normal" and "abnormal" rather than specific types of defects, completely eliminating any form of training or offline modeling stage. Therefore, for completely new defect types that have never appeared in the training set, as long as their feature representations differ from normal regions, this method can detect them, thus possessing strong generalization ability.

[0026] It should be noted that, in order to preserve the pre-trained knowledge and maintain the consistency of the extracted features, the network weights of the deep convolutional neural network in this embodiment are fixed and no fine-tuning based on the target data is performed. The network's general feature representation capabilities learned on large datasets are utilized to avoid overfitting or knowledge forgetting that may be caused by fine-tuning on small-scale or specific target datasets. It is used as a fixed feature extractor. After inputting the standard image and the image to be tested, the features are extracted through forward propagation and compared to output the results, achieving the purpose of plug and play.

[0027] Preferably, this embodiment uses a deep convolutional neural network ResNet-50 as an example. The deep convolutional neural network includes an input layer and several consecutive residual stages. The input layer includes convolutional layers and pooling layers. The convolutional kernel size of the convolutional layer is 7x7, the stride is 2, and the number of output channels is 64. The convolutional layer maps a single-channel grayscale image to a 64-channel feature map, capturing basic edge and texture features. At the same time, the stride of 2 achieves the first downsampling, reducing the input image size to 1 / 2. The pooling kernel size of the pooling layer is 3x3, and the stride is also 2. The pooling layer further downsamples, reducing the input image size to 1 / 2 again, enhancing the translation invariance of features, and filtering local noise.

[0028] Each residual stage is composed of stacked residual blocks, and each residual block contains at least a convolutional layer, a batch normalization layer, and an activation function. In this embodiment, four consecutive residual stages are used: layer1, layer2, layer3, and layer4. Layer1 contains 3 residual blocks, with 256 output feature map channels and a large spatial size, preserving rich detail information. Layer2 contains 4 residual blocks, with 512 output channels and a halved spatial size. Layer3 contains 6 residual blocks, with 1024 output channels and a halved spatial size again, capturing mid-level semantic features. Layer4 contains 3 residual blocks, with 2048 output channels and the smallest spatial size, containing highly abstract high-level semantic features.

[0029] In this step, since the deep convolutional neural network only utilizes the feature maps of the intermediate layers, the output layer of the deep convolutional neural network is discarded. Simultaneously, a multi-output model is created to achieve parallel multi-level feature extraction, inputting the image into this reconstruction model all at once. During one forward propagation cycle of the network, the input data sequentially passes through all layers, capturing the output tensors at multiple specified intermediate layer locations; these tensors are the required multi-scale feature maps. To achieve multi-scale coverage from details to intermediate semantics to deep semantics, ensuring that wafer defects of different types and sizes can be captured, this embodiment selects several residual stages: shallow residual stage, intermediate residual stage, and deep residual stage, namely shallow residual stage layer1, intermediate residual stage layer3, and deep residual stage layer4, and captures the feature maps output by these residual stages.

[0030] S03. Based on the feature map, calculate the probability vectors of the same spatial locations in the standard image and the image to be inspected, calculate the divergence value based on the probability vectors, and generate a difference map.

[0031] For the standard feature map of the standard image and the feature map to be detected of the image output in the same residual stage in step S02, calculate the difference between the standard feature map and the feature map to be detected in different residual stages. The calculation method includes, but is not limited to, the Kullback-Leibler divergence used in this embodiment. Other metrics such as Jensen-Shannon divergence, Euclidean distance (L2Distance), cosine distance (Cosine Distance), or correlation coefficient can also be used. The specific calculation process is as follows: Based on the standard feature map of the standard image output from the same residual stage and the feature map of the image to be inspected, for each spatial location... Extract the C-dimensional standard feature vector from the standard feature map. and the C-dimensional feature vector in the feature map to be detected. The standard feature vector and the feature vector to be detected are converted into standard probability vectors using a transformation function. and the probability vector to be detected The transformation function used in this embodiment is Softmax:

[0032]

[0033] The formula for calculating the Softmax function is as follows: Ensure that the sum of all elements is 1.

[0034] After obtaining the standard probability vector and the probability vector to be detected, for each spatial location Calculate the divergence value of the probability vector to be detected mapped to the standard probability vector, and sum it over all channels:

[0035] in, It is the probability value of the standard probability vector distribution on the c-th channel. It is the natural logarithm of the probability value. It is the natural logarithm of the probability values ​​of the probability vector to be detected distributed on the c-th channel.

[0036] In actual calculations, to prevent taking the logarithm of zero ( This leads to numerical errors. Before calculating the logarithm, a very small positive number epsilon (ε, e.g., 1e-8) is applied to the probability value to clamp it, i.e.: The actual calculation is .

[0037] Traversing each spatial location in the feature map Calculate the corresponding KL divergence value according to the above formula. The calculation results for all locations are filled into a two-dimensional matrix according to their original spatial coordinates, ultimately generating a difference map with the same spatial dimensions as the feature map of that layer. Each pixel value in this map directly quantifies the degree of feature dissimilarity between the input image pairs in the corresponding local region. The generation process of can be uniformly represented by the following mathematical transformations:

[0038]

[0039]

[0040] Collect difference maps at different levels obtained from the selected residual stages to obtain a set of multi-scale difference representations.

[0041] S04. Merge the difference maps at different levels to obtain a comprehensive difference map, perform post-processing on the comprehensive difference map, and output a binary defect location map for automatic optical inspection.

[0042] Since the difference maps at different levels in step S03 have different spatial resolutions, it is necessary to upsample the low-resolution difference maps to restore the size of several difference maps within the difference representation to the same size as the image to be inspected. Then, these restored difference maps are fused into a comprehensive difference map using a weighted summation method. The weight coefficients in the weighted summation process can be manually assigned based on the importance of features at different levels or determined through a simple optimization strategy. Alternatively, the fusion method in this step can also employ a multi-scale image fusion algorithm (such as Laplacian pyramid fusion) or utilize a simple small convolutional neural network for adaptive fusion.

[0043] The obtained composite difference map is post-processed to generate the final binarized defect localization map. Specifically, an adaptive thresholding method is used to binarize the composite difference map to obtain the optimal threshold, or an empirical fixed threshold is set to binarize the difference map. Based on the optimal threshold or the fixed threshold, areas with pixel values ​​greater than the optimal threshold or the fixed threshold are identified as potential defect areas. Morphological operations are applied to the binarized composite difference map, such as opening and closing operations. Opening operations can remove small noise points and fill holes in the defect area, while closing operations can fill holes in the defect area and smooth its contour, ultimately obtaining a connected, denoised defect binary mask.

[0044] The system outputs a binary defect location map, where white areas (high pixel values) mark detected abnormal locations, and black areas (low pixel values) represent the normal background. This system can be integrated into AOI equipment to automatically trigger alarms, recording, or sorting mechanisms, completing an automated inspection process.

[0045] In summary, the zero-shot wafer defect detection method based on multi-scale feature difference comparison provided by this invention completely eliminates the training or modeling stage. It employs a fixed, pre-trained deep convolutional neural network on a general dataset as the feature extractor. Zero-shot anomaly detection and localization are achieved by directly calculating the differences between the test image and a defect-free reference image in a multi-scale deep feature space. Feature maps from multiple layers of the pre-trained network are extracted, and a metric function is used to calculate the difference values ​​between corresponding feature maps layer by layer, generating multiple difference maps. These multiple difference maps at different resolutions are upsampled to the same scale and weighted and fused to generate a comprehensive difference map with significant anomalies for automatic optical inspection. This invention directly uses a general, frozen-weight pre-trained model as the feature extractor, obtaining results through a single forward propagation and efficient feature difference calculation, achieving true plug-and-play functionality. Furthermore, it only requires one or a few golden reference images to operate, greatly reducing data dependence, simplifying the deployment process, shortening the deployment cycle, and lowering application costs.

[0046] Example 2 In another aspect, this invention provides a zero-sample wafer defect detection system based on multi-scale feature difference comparison. Please refer to [link / reference]. Figure 2 The diagram shown is a structural schematic of the zero-sample wafer defect detection system for multi-scale feature difference comparison in Embodiment 2 of the present invention. The zero-sample wafer defect detection system for multi-scale feature difference comparison includes: Alignment module 11 is used to acquire the preprocessed standard image and the image to be inspected, and to align the standard image and the image to be inspected in space; Extraction module 12 is used to construct a pre-trained deep convolutional neural network, fix the network weights of the deep convolutional neural network, use the deep convolutional neural network to perform multi-level feature extraction on the standard image and the image to be inspected, and obtain the feature map of each level respectively. Calculation module 13 is used to calculate the probability vectors of the same spatial locations in the standard image and the image to be inspected based on the feature map, calculate the divergence value based on the probability vectors, and generate a difference map. Output module 14 is used to fuse difference maps at different levels to obtain a comprehensive difference map, perform post-processing on the comprehensive difference map, and output a binarized defect location map for automatic optical inspection.

[0047] Obtain a known gold reference image as the standard image, and an image to be inspected. Perform necessary preprocessing on the standard image and the image to be inspected. This preprocessing includes, but is not limited to, image size normalization, color space conversion (such as converting to grayscale), and fine image registration to ensure that the two images are perfectly aligned in space.

[0048] In this embodiment, the constructed deep convolutional neural network needs to learn general and powerful feature representations in advance on millions of natural images. These features have high semantic information and can effectively distinguish between "normal" and "abnormal" rather than specific types of defects, completely eliminating any form of training or offline modeling stage. Therefore, for completely new defect types that have never appeared in the training set, as long as their feature representations differ from normal regions, this method can detect them, thus possessing strong generalization ability.

[0049] It should be noted that, in order to preserve the pre-trained knowledge and maintain the consistency of the extracted features, the network weights of the deep convolutional neural network in this embodiment are fixed and no fine-tuning based on the target data is performed. The network's general feature representation capabilities learned on large datasets are utilized to avoid overfitting or knowledge forgetting that may be caused by fine-tuning on small-scale or specific target datasets. It is used as a fixed feature extractor. After inputting the standard image and the image to be tested, the features are extracted through forward propagation and compared to output the results, achieving the purpose of plug and play.

[0050] Preferably, this embodiment uses a deep convolutional neural network ResNet-50 as an example. The deep convolutional neural network includes an input layer and several consecutive residual stages. The input layer includes convolutional layers and pooling layers. The convolutional kernel size of the convolutional layer is 7x7, the stride is 2, and the number of output channels is 64. The convolutional layer maps a single-channel grayscale image to a 64-channel feature map, capturing basic edge and texture features. At the same time, the stride of 2 achieves the first downsampling, reducing the input image size to 1 / 2. The pooling kernel size of the pooling layer is 3x3, and the stride is also 2. The pooling layer further downsamples, reducing the input image size to 1 / 2 again, enhancing the translation invariance of features, and filtering local noise.

[0051] Each residual stage is composed of stacked residual blocks, and each residual block contains at least a convolutional layer, a batch normalization layer, and an activation function. In this embodiment, four consecutive residual stages are used: layer1, layer2, layer3, and layer4. Layer1 contains 3 residual blocks, with 256 output feature map channels and a large spatial size, preserving rich detail information. Layer2 contains 4 residual blocks, with 512 output channels and a halved spatial size. Layer3 contains 6 residual blocks, with 1024 output channels and a halved spatial size again, capturing mid-level semantic features. Layer4 contains 3 residual blocks, with 2048 output channels and the smallest spatial size, containing highly abstract high-level semantic features.

[0052] In this step, since the deep convolutional neural network only utilizes the feature maps of the intermediate layers, the output layer of the deep convolutional neural network is discarded. Simultaneously, a multi-output model is created to achieve parallel multi-level feature extraction, inputting the image into this reconstruction model all at once. During one forward propagation cycle of the network, the input data sequentially passes through all layers, capturing the output tensors at multiple specified intermediate layer locations; these tensors are the required multi-scale feature maps. To achieve multi-scale coverage from details to intermediate semantics to deep semantics, ensuring that wafer defects of different types and sizes can be captured, this embodiment selects several residual stages: shallow residual stage, intermediate residual stage, and deep residual stage, namely shallow residual stage layer1, intermediate residual stage layer3, and deep residual stage layer4, and captures the feature maps output by these residual stages.

[0053] For the standard feature map of the standard image and the feature map to be detected of the image output in the same residual stage in step S02, calculate the difference between the standard feature map and the feature map to be detected in different residual stages. The calculation method includes, but is not limited to, the Kullback-Leibler divergence used in this embodiment. Other metrics such as Jensen-Shannon divergence, Euclidean distance (L2Distance), cosine distance (Cosine Distance), or correlation coefficient can also be used. The specific calculation process is as follows: Based on the standard feature map of the standard image output from the same residual stage and the feature map of the image to be inspected, for each spatial location... Extract the C-dimensional standard feature vector from the standard feature map. and the C-dimensional feature vector in the feature map to be detected. The standard feature vector and the feature vector to be detected are converted into standard probability vectors using a transformation function. and the probability vector to be detected The transformation function used in this embodiment is Softmax:

[0054]

[0055] The formula for calculating the Softmax function is as follows: Ensure that the sum of all elements is 1.

[0056] After obtaining the standard probability vector and the probability vector to be detected, for each spatial location Calculate the divergence value of the probability vector to be detected mapped to the standard probability vector, and sum it over all channels:

[0057] in, It is the probability value of the standard probability vector distribution on the c-th channel. It is the natural logarithm of the probability value. It is the natural logarithm of the probability values ​​of the probability vector to be detected distributed on the c-th channel.

[0058] In actual calculations, to prevent taking the logarithm of zero ( This leads to numerical errors. Before calculating the logarithm, a very small positive number epsilon (ε, e.g., 1e-8) is applied to the probability value to clamp it, i.e.: The actual calculation is .

[0059] Traversing each spatial location in the feature map Calculate the corresponding KL divergence value according to the above formula. The calculation results for all locations are filled into a two-dimensional matrix according to their original spatial coordinates, ultimately generating a difference map with the same spatial dimensions as the feature map of that layer. Each pixel value in this map directly quantifies the degree of feature dissimilarity between the input image pairs in the corresponding local region. The generation process of can be uniformly represented by the following mathematical transformations:

[0060]

[0061]

[0062] Collect difference maps at different levels obtained from the selected residual stages to obtain a set of multi-scale difference representations.

[0063] Since the difference maps at different levels in step S03 have different spatial resolutions, it is necessary to upsample the low-resolution difference maps to restore the size of several difference maps within the difference representation to the same size as the image to be inspected. Then, these restored difference maps are fused into a comprehensive difference map using a weighted summation method. The weight coefficients in the weighted summation process can be manually assigned based on the importance of features at different levels or determined through a simple optimization strategy. Alternatively, the fusion method in this step can also employ a multi-scale image fusion algorithm (such as Laplacian pyramid fusion) or utilize a simple small convolutional neural network for adaptive fusion.

[0064] The obtained composite difference map is post-processed to generate the final binarized defect localization map. Specifically, an adaptive thresholding method is used to binarize the composite difference map to obtain the optimal threshold, or an empirical fixed threshold is set to binarize the difference map. Based on the optimal threshold or the fixed threshold, areas with pixel values ​​greater than the optimal threshold or the fixed threshold are identified as potential defect areas. Morphological operations are applied to the binarized composite difference map, such as opening and closing operations. Opening operations can remove small noise points and fill holes in the defect area, while closing operations can fill holes in the defect area and smooth its contour, ultimately obtaining a connected, denoised defect binary mask.

[0065] The system outputs a binary defect location map, where white areas (high pixel values) mark detected abnormal locations, and black areas (low pixel values) represent the normal background. This system can be integrated into AOI equipment to automatically trigger alarms, recording, or sorting mechanisms, completing an automated inspection process.

[0066] In summary, the zero-shot wafer defect detection system based on multi-scale feature difference comparison provided by this invention completely eliminates the training or modeling stage, employing a fixed, pre-trained deep convolutional neural network on a general dataset as the feature extractor. It achieves zero-shot anomaly detection and localization by directly calculating the differences between the test image and a defect-free reference image in a multi-scale deep feature space. It extracts feature maps from multiple layers of the pre-trained network and uses a metric function to calculate the difference values ​​between corresponding feature maps layer by layer, generating multiple difference maps. These multiple difference maps at different resolutions are upsampled to the same scale and weighted and fused to generate a comprehensive difference map with significant anomalies for automatic optical inspection. This invention directly uses a general, frozen-weight pre-trained model as the feature extractor, obtaining results through a single forward propagation and efficient feature difference calculation, achieving true plug-and-play functionality. Furthermore, it only requires one or a few golden reference images to operate, greatly reducing data dependence, simplifying the deployment process, shortening the deployment cycle, and lowering application costs.

[0067] Example 3 In another aspect, the present invention also proposes a computer-readable storage medium having stored thereon one or more computer programs that, when executed by a processor, implement the above-described zero-sample wafer defect detection method for multi-scale feature difference comparison.

[0068] Those skilled in the art will understand that the logic or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable storage medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable storage medium" can mean any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0069] More specific examples (a non-exhaustive list) of computer-readable storage media include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable storage media can even be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0070] Example 4 Figure 3 This is a structural block diagram of an electronic device provided in Embodiment 4. The electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the zero-sample wafer defect detection method for multi-scale feature difference comparison described in the above embodiments. Figure 3 The electronic device 30 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of the present invention.

[0071] like Figure 3 As shown, the electronic device 30 can be manifested as a general-purpose computing device, such as a server device. The components of the electronic device 30 may include, but are not limited to: at least one processor 31, at least one memory 32, and a bus 33 connecting different system components (including memory 32 and processor 31).

[0072] Bus 33 includes a data bus, an address bus, and a control bus.

[0073] The memory 32 may include volatile memory, such as RAM 321 (random access memory), and / or cache memory 322, and may further include ROM 323 (read-only memory).

[0074] The memory 32 may also include a program tool 325 having a set (at least one) of program modules 324, including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0075] The processor 31 executes various functional applications and data processing by running computer programs stored in the memory 32, such as the zero-sample wafer defect detection method for multi-scale feature difference comparison as described above.

[0076] Electronic device 30 can also communicate with one or more external devices 34 (e.g., keyboard, pointing device, etc.). This communication can be performed via I / O interface 35 (input / output interface). Furthermore, electronic device 30 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public network, such as the Internet) via network adapter 36. Figure 3 As shown, network adapter 36 communicates with other modules of the model-generated electronic device 30 via bus 33. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with the model-generated electronic device 30, including but not limited to: microcode, device drivers, redundant processors, disk drive arrays, RAID (disk array) systems, tape drives, and data backup storage systems.

[0077] It should be noted that although several units / modules or sub-units / modules of the electronic device have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of the present invention, the features and functions of two or more units / modules described above can be embodied in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided and embodied by multiple units / modules.

[0078] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0079] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

Claims

1. A multi-scale feature difference comparison based zero-shot wafer defect detection method, characterized in that, The zero-sample wafer defect detection method based on multi-scale feature difference comparison comprises the following steps: acquiring a preprocessed standard image and a to-be-detected image, and spatially aligning the standard image and the to-be-detected image; constructing a pre-trained deep convolutional neural network, fixing the network weight of the deep convolutional neural network, using the deep convolutional neural network to perform multi-level feature extraction on the standard image and the to-be-detected image, and acquiring a feature map of each level respectively; based on the feature map, calculating a probability vector of the same spatial position of the standard image and the to-be-detected image respectively, calculating a divergence value according to the probability vector, and generating a difference map; fusing difference maps of different levels to acquire a comprehensive difference map, post-processing the comprehensive difference map, and outputting a binary defect positioning map for automatic optical detection.

2. The method of claim 1, wherein the multi-scale feature difference comparison is performed by a convolutional neural network (CNN). The step of constructing a pre-trained deep convolutional neural network comprises the following steps: ​ constructing a pre-trained deep convolutional neural network, the deep convolutional neural network comprising an input layer and a plurality of continuous residual stages, the input layer comprising a convolutional layer and a pooling layer, and any residual stage being stacked by a plurality of residual blocks, any residual block at least containing a convolutional layer, a batch normalization layer, and an activation function.

3. The method of claim 2, wherein the multi-scale feature difference comparison is performed by a convolutional neural network (CNN). The step of using the deep convolutional neural network to perform multi-level feature extraction on the standard image and the to-be-detected image and acquiring a feature map of each level respectively comprises the following steps: ​ inputting the standard image and the to-be-detected image into the deep convolutional neural network, selecting shallow residual stages, middle residual stages, and deep residual stages in the plurality of residual stages in one forward propagation period of the deep convolutional neural network, and capturing feature maps output by the selected residual stages in the selected residual stages.

4. The zero-sample wafer defect detection method based on multi-scale feature difference comparison according to claim 1, characterized in that, The step of calculating a probability vector of the same spatial position of the standard image and the to-be-detected image respectively based on the feature map, calculating a divergence value according to the probability vector, and generating a difference map comprises the following steps: Based on the standard feature map of the standard image output from the same residual stage and the feature map of the image to be inspected, for each spatial location... Extract the C-dimensional standard feature vector from the standard feature map. and the C-dimensional feature vector in the feature map to be detected. The standard feature vector and the feature vector to be detected are converted into a standard probability vector and a probability vector to be detected, respectively, using a conversion function.

5. The method of claim 4, wherein the multi-scale feature difference comparison is performed by a convolutional neural network (CNN). 5 After obtaining the standard probability vector and the to-be-tested probability vector, for each spatial position , the divergence value of the to-be-tested probability vector mapping on the standard probability vector is calculated, and the sum of all channels is obtained: wherein is the probability value of the standard probability vector distribution on the cth channel, is the natural logarithm of the probability value, is the natural logarithm of the probability value of the probability vector distribution to be examined on the cth channel.

6. The method of claim 5, wherein the multi-scale feature difference comparison is performed by a convolutional neural network (CNN). 5 The step of generating a difference map comprises the following steps: According to the standard feature map and each spatial position in the to-be-detected feature map The spatial position The calculated divergence value is filled in a matrix of a preset size based on the spatial coordinates of the spatial position, wherein the size of the matrix is consistent with the spatial size of the feature map; collecting difference maps of different levels calculated by the selected residual stages to acquire a group of multi-scale difference representations.

7. The method of claim 1, wherein the multi-scale feature difference comparison is performed by a convolutional neural network (CNN) trained on a dataset of labeled training images. The step of fusing difference maps of different levels to acquire a comprehensive difference map, post-processing the comprehensive difference map, and outputting a binary defect positioning map for automatic optical detection comprises the following steps: restoring the size of a plurality of difference maps in the difference representation to the same size as the to-be-detected image, and fusing the plurality of difference maps with restored size into a comprehensive difference map; applying an adaptive threshold method to the comprehensive difference map to perform binaryzation, acquiring an optimal threshold value of the comprehensive difference map, marking a defect region in the comprehensive difference map according to the optimal threshold value, applying a morphological operation to the binaryzated comprehensive difference map to remove tiny noise points and fill holes in the defect region, and outputting a binary defect positioning map for automatic optical detection.

8. A multi-scale feature difference-based zero-shot wafer defect detection system, comprising: The zero-sample wafer defect detection system based on multi-scale feature difference comparison is used to implement the zero-sample wafer defect detection method based on multi-scale feature difference comparison according to any one of claims 1-7, and the system comprises the following components: An alignment module is configured to acquire a preprocessed standard image and a to-be-detected image, and align the standard image and the to-be-detected image in space; An extraction module is configured to construct a pre-trained deep convolutional neural network, fix network weights of the deep convolutional neural network, perform multi-level feature extraction on the standard image and the to-be-detected image by using the deep convolutional neural network, and acquire a feature map of each level respectively; A calculation module is configured to calculate a probability vector of the same spatial position of the standard image and the to-be-detected image respectively based on the feature map, calculate a divergence value according to the probability vector, and generate a difference map; An output module is configured to fuse difference maps of different levels, acquire a comprehensive difference map, perform post-processing on the comprehensive difference map, and output a binary defect positioning map for automatic optical detection.

9. A computer readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to implement the multi-scale feature difference comparison zero-sample wafer defect detection method of any one of claims 1-7.

10. An electronic device comprising a memory, a processor, and a computer program stored on the memory and running on the processor, characterized in that, The processor executes the computer program to implement the multi-scale feature difference comparison zero-sample wafer defect detection method of any one of claims 1-7.

Images