Wafer defect detection method for resisting auto-encoder based on multi-modal feature fusion

Abstract

The invention discloses an anti-auto-encoder wafer defect detection method based on multi-modal feature fusion, which is used in the technical field of wafer defect detection and comprises the following steps: acquiring an image of a wafer to be detected; preprocessing the to-be-detected wafer image to obtain a preprocessed to-be-detected wafer image; inputting the preprocessed image of the wafer to be detected into the trained multi-modal Wasserstein auto-encoder, and performing forward propagation by using a multi-modal feature encoder and a multi-modal classifier of the trained multi-modal Wasserstein auto-encoder to obtain probability distribution of each defect type; and selecting the defect type corresponding to the maximum probability as a final defect classification result. According to the wafer defect detection method, the convergence process is relatively stable, the requirement for the labeled data volume is reduced, the spatial features, the frequency domain features and the texture features are fused, and the classification accuracy is high.

Description

Technical Field

[0001] This invention belongs to the field of wafer defect detection technology, specifically relating to a wafer defect detection method based on multimodal feature fusion and an adversarial autoencoder. Background Technology

[0002] As semiconductor manufacturing processes continue to evolve to the nanometer scale, the morphology of wafer defects is becoming increasingly complex and their size is becoming smaller, posing a severe challenge to the final yield of chips.

[0003] Modern production lines generate a large amount of wafer image data daily, including various defect patterns such as Center (defects concentrated in the wafer's central area), Donut (ring-shaped defects, with a intact center but damaged periphery), Edge-Loc (defects located in a localized area at the wafer's edge), Edge-Ring (defects distributed in a complete ring along the wafer's edge), Loc (randomly distributed localized defects), Near-full (defects almost covering the entire wafer), Random (randomly distributed point defects), and Scratch (linear scratch defects). Traditional methods relying on manual visual inspection suffer from inherent bottlenecks such as low efficiency, high subjectivity, and high cost, and can no longer meet the stringent requirements of intelligent manufacturing for inspection speed and accuracy.

[0004] However, existing deep learning-based methods generally face core challenges such as scarce labeled data, unbalanced number of defect category samples, and single feature representation dimension, which restrict their further application and performance improvement in real industrial scenarios. Summary of the Invention

[0005] To address the aforementioned technical problems in the existing technology, this invention provides a wafer defect detection method based on multimodal feature fusion and an adversarial autoencoder. The technical problem to be solved by this invention is achieved through the following technical solution: This invention provides a wafer defect detection method based on multimodal feature fusion and adversarial autoencoder, the wafer defect detection method comprising: Acquire an image of the wafer to be inspected; The image of the wafer to be inspected is preprocessed to obtain a preprocessed image of the wafer to be inspected; The preprocessed wafer image to be inspected is input into the trained multimodal Wasserstein autoencoder. The multimodal feature encoder and multimodal classifier of the trained multimodal Wasserstein autoencoder are used to perform forward propagation to obtain the probability distribution of each defect type, and the defect type corresponding to the highest probability is selected as the final defect classification result. The multimodal feature encoder is used to obtain a first fusion feature based on the first spatial feature, the first frequency domain feature, and the first texture feature extracted from the preprocessed wafer image to be inspected, and to obtain a latent feature based on the first fusion feature. The multimodal classifier is used to obtain the probability distribution of each defect type based on the latent feature.

[0006] In an optional embodiment, the wafer image to be inspected is preprocessed to obtain a preprocessed wafer image to be inspected, including: The wafer image to be inspected is subjected to size normalization and pixel value normalization in sequence to obtain the preprocessed wafer image to be inspected.

[0007] In an optional embodiment, the preprocessed wafer image to be inspected is input into a trained multimodal Wasserstein autoencoder. The multimodal feature encoder and multimodal classifier of the trained multimodal Wasserstein autoencoder are forward-propagated to obtain the probability distribution of each defect type. The defect type corresponding to the highest probability is selected as the final defect classification result, including: The preprocessed wafer image to be inspected is input into the multimodal feature encoder to extract the first spatial feature, the first frequency domain feature, and the first texture feature from the preprocessed wafer image to be inspected. The first fusion feature is obtained based on the first spatial feature, the first frequency domain feature, and the first texture feature; The first fused feature is mapped to the latent space to obtain the latent feature; The latent features are input into the multimodal classifier to obtain the probability distribution of each defect type; The defect type with the highest probability from the probability distribution of each defect type is selected as the final defect classification result.

[0008] In an optional embodiment, the preprocessed wafer image to be inspected is input into the multimodal feature encoder to extract the first spatial feature, the first frequency domain feature, and the first texture feature from the preprocessed wafer image to be inspected, including: The preprocessed wafer image to be inspected is input into the spatial feature extraction module of the multimodal feature encoder. The preprocessed wafer image to be inspected is processed by a three-level convolutional downsampling network to obtain downsampled features. The downsampled features are flattened and input into the first fully connected layer, and mapped to the first spatial features by the first fully connected layer. The three-level convolutional downsampling network includes a first convolutional layer, a first activation function, a first downsampling layer, a second convolutional layer, a second activation function, a second downsampling layer, a third convolutional layer, a third activation function, and a third downsampling layer. The preprocessed wafer image to be detected is input into the frequency domain feature extraction module of the multimodal feature encoder. The preprocessed wafer image to be detected is passed through the fourth convolutional layer to obtain four frequency band features. The four frequency band features are respectively subjected to global average pooling to obtain four global average pooling features. The four global average pooling features are simultaneously input into two second fully connected layers to obtain four frequency band attention weights. Each frequency band attention weight is weighted with the corresponding frequency band feature to obtain four enhanced frequency band features. Each enhanced frequency band feature is subjected to adaptive pooling compression to obtain adaptive pooling compressed frequency band features. The four adaptive pooling compressed frequency band features are flattened into the first frequency domain features. The preprocessed wafer image to be detected is input into the texture feature extraction module of the multimodal feature encoder. The preprocessed wafer image to be detected is processed by a three-level convolutional network to obtain convolutional features. The convolutional features are passed through two third fully connected layers to obtain attention weights for each channel of the convolutional features. The attention weights of each channel are weighted with the convolutional features of the corresponding channel to obtain weighted features for each channel. Each weighted feature is compressed by adaptive pooling to obtain compressed weighted features. The compressed weighted features of each channel are flattened to form the first texture feature. The three-level convolutional network includes a fifth convolutional layer, a fourth activation function, a sixth convolutional layer, a fifth activation function, a seventh convolutional layer, and a sixth activation function.

[0009] In an optional embodiment, the first fused feature is obtained based on the first spatial feature, the first frequency domain feature, and the first texture feature, including: The first spatial feature, the first frequency domain feature, and the first texture feature are projected onto a unified feature space of the same dimension through the fourth fully connected layer, the fifth fully connected layer, and the sixth fully connected layer, respectively, to obtain the second spatial feature, the second frequency domain feature, and the second texture feature of the same dimension. The second spatial feature, the second frequency domain feature, and the second texture feature are stacked into a sequence consisting of three elements; By utilizing a multi-head attention mechanism, three elements in the sequence are made to interact across modalities to obtain a third spatial feature, a third frequency domain feature, and a third texture feature. The second spatial feature, the second frequency domain feature, and the second texture feature are respectively coupled with the third spatial feature, the third frequency domain feature, and the third texture feature to obtain the fourth spatial feature, the fourth frequency domain feature, and the fourth texture feature; The fourth spatial feature, the fourth frequency domain feature, and the fourth texture feature are concatenated to obtain the concatenated feature; The concatenated features are input into a gated fusion network to obtain three modal weights. The first fused feature is obtained by weighting and summing the three modal weights with the corresponding fourth spatial feature, fourth frequency domain feature, and fourth texture feature.

[0010] In an optional embodiment, the latent features are input into the multimodal classifier to obtain the probability distribution of each defect type, including: The potential features are processed through spatial branching, frequency domain branching, and texture branching respectively to obtain the fifth spatial feature, the fifth frequency domain feature, and the fifth texture feature; The second fused feature is obtained by weighted summation of the fifth spatial feature, the fifth frequency domain feature, and the fifth texture feature; The probability distribution of each defect type is obtained based on the second fusion feature.

[0011] In an optional embodiment, the multimodal Wasserstein autoencoder further includes a feature reconstruction decoder; The training method for the multimodal Wasserstein autoencoder includes: Obtain a labeled training dataset and an unlabeled training dataset. The labeled training dataset includes multiple preprocessed images of the wafer to be trained and corresponding defect type labels. The unlabeled training dataset includes multiple preprocessed images of the wafer to be trained. The labeled training dataset and the unlabeled training dataset are divided into multiple batches and sequentially input into the multimodal Wasserstein autoencoder. The Adam optimizer is used, combined with gradient pruning technology, to perform backpropagation and parameter update on the multi-objective loss function, resulting in a trained multimodal Wasserstein autoencoder.

[0012] In an optional embodiment, the feature reconstruction decoder is used to receive the latent features to be trained output by the multimodal feature encoder. The latent features to be trained are passed through a thirteenth fully connected layer to obtain extended features. The extended features are then upsampled through three transposed convolutional layers to obtain a reconstructed image.

[0013] In an optional embodiment, the multi-objective loss function is expressed as: ; in, For multi-target loss, To reconstruct the loss, For distributed matching loss, For classifying losses, For modal consistency loss, , , These are preset weighting coefficients; For the training wafer images in the unlabeled training dataset, the classification loss and modality consistency loss in the multi-objective loss function are both set to 0.

[0014] In an optional embodiment, the reconstruction loss, the distribution matching loss, the classification loss, and the modality consistency loss are respectively expressed as: ; ; ; ; in, The number of samples in each batch. This represents the total number of defect types. For the first i One sample, For the first i A reconstructed image, For the first i One potential feature to be trained For the first j One potential feature to be trained The first sampled from the standard normal distribution i A random vector, The first sampled from the standard normal distribution j A random vector, It is an inverse multivariate quadratic kernel function. For the first c Weighting coefficients for each defect type For the first i The sample at the th c The true label on each defect type For the predicted first i The sample belongs to the first c The probability of each defect type The modal weights output by the gated fusion network of the multimodal feature encoder. The weights of each branch of the multimodal classifier are given. For variance, Standard deviation, This is the mean.

[0015] Compared with the prior art, the beneficial effects of the present invention are as follows: The wafer defect detection method provided by this invention utilizes a trained multimodal Wasserstein autoencoder for wafer defect detection. The multimodal feature encoder of the multimodal Wasserstein autoencoder obtains a first fused feature based on the first spatial feature, first frequency domain feature, and first texture feature extracted from the preprocessed wafer image to be inspected. Based on the first fused feature, latent features are obtained. The multimodal classifier of the multimodal Wasserstein autoencoder then obtains the probability distribution of each defect type based on the latent features. This wafer defect detection method exhibits a relatively stable convergence process, reduces the requirement for labeled data, and integrates spatial, frequency, and texture features with high classification accuracy. It helps improve the efficiency of quality inspection in semiconductor manufacturing processes and has a positive effect on the synergistic optimization of multimodal feature fusion and semi-supervised learning.

[0016] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0017] Figure 1 This is a flowchart of a wafer defect detection method based on multimodal feature fusion and adversarial autoencoder provided in an embodiment of the present invention; Figure 2 This is a classification performance graph of the WM811K dataset provided in an embodiment of the present invention. Detailed Implementation

[0018] To further illustrate the technical means and effects adopted by the present invention to achieve the intended purpose, the following describes in detail, with reference to the accompanying drawings and specific embodiments, a wafer defect detection method based on multimodal feature fusion and adversarial autoencoder proposed according to the present invention.

[0019] The foregoing and other technical contents, features, and effects of the present invention will be clearly presented in the following detailed description of specific embodiments in conjunction with the accompanying drawings. Through the description of the specific embodiments, a more in-depth and concrete understanding can be gained of the technical means and effects adopted by the present invention to achieve its intended purpose. However, the accompanying drawings are for reference and illustration only and are not intended to limit the technical solutions of the present invention.

[0020] Currently, traditional machine learning methods have significant limitations in addressing wafer defect classification problems. Schemes based on manual feature extraction heavily rely on the quality of feature engineering and the completeness of prior domain knowledge. When faced with the complex and varied defect morphologies at the nanoscale, they suffer from insufficient feature representation and limited generalization ability.

[0021] While existing deep learning methods can automatically learn feature representations, they generally rely on large-scale, high-quality labeled data. In real-world semiconductor manufacturing scenarios, labeling defect samples is costly, and the number of samples for different defect categories is severely imbalanced, making it difficult for models to meet the accuracy requirements of industrial-grade testing. Consequently, the accuracy for identifying a few types of defects (such as Scratch, Loc, etc.) is low.

[0022] Current semi-supervised methods based on autoencoders have shortcomings in latent space modeling: traditional autoencoders lack explicit constraints on the latent distribution, resulting in weak discriminative power in representation learning; while variational autoencoders, although introducing probabilistic modeling, are prone to posterior collapse during training, leading to a decline in the expressive power of latent features. Furthermore, most existing methods extract features only from a single image spatial domain, failing to effectively integrate multimodal defect information such as frequency domain and texture, thus limiting the model's comprehensive perception and identification of the essential characteristics of defects. Regarding data utilization, existing semi-supervised methods still have room for improvement in their efficiency in utilizing unlabeled data, failing to fully explore their potential in high-dimensional feature learning and distribution modeling.

[0023] Therefore, please see Figure 1 , Figure 1 This is a flowchart of a wafer defect detection method based on multimodal feature fusion and adversarial autoencoder provided by an embodiment of the present invention. The present invention proposes a wafer defect detection method based on multimodal feature fusion and adversarial autoencoder, which includes: Step 1: Obtain an image of the wafer to be inspected.

[0024] Here, a wafer image is a distribution map formed by spatially mapping the test results of each chip on the wafer according to its actual position on the wafer after testing each chip on the wafer with electrical testing equipment. The wafer image to be inspected is the wafer image that needs to be inspected for defects.

[0025] Step 2: Preprocess the image of the wafer to be inspected to obtain the preprocessed image of the wafer to be inspected.

[0026] In one specific embodiment, step 2 may include: The wafer image to be inspected is subjected to size normalization and pixel value normalization in sequence to obtain the preprocessed wafer image to be inspected.

[0027] Specifically, the wafer image to be inspected is first normalized to a size of 32×32, and then pixel value normalized to a pixel value in the range of [-1, 1], thus obtaining the preprocessed wafer image to be inspected.

[0028] Step 3: Input the preprocessed wafer image to be inspected into the trained multimodal Wasserstein autoencoder. Use the multimodal feature encoder and multimodal classifier of the trained multimodal Wasserstein autoencoder to perform forward propagation to obtain the probability distribution of each defect type, and select the defect type corresponding to the highest probability as the final defect classification result.

[0029] Specifically, the multimodal Wasserstein autoencoder in this embodiment includes a multimodal feature encoder, a feature reconstruction decoder, and a multimodal classifier. During training, the multimodal feature encoder, feature reconstruction decoder, and multimodal classifier need to be trained to obtain a trained multimodal Wasserstein autoencoder. When the trained multimodal Wasserstein autoencoder is used to process the preprocessed wafer image to be inspected, the multimodal feature encoder and multimodal classifier of the trained multimodal Wasserstein autoencoder are used for forward propagation to obtain the probability distribution of each defect type. The defect type with the highest probability is selected from the probabilities of each defect type as the final defect classification result of the wafer image to be inspected. The defect types are divided into Center, Donu, Edge-Loc, Edge-Ring, Loc, Near-full, Random, Scratch, and None (defect-free normal wafer).

[0030] In this embodiment, the multimodal feature encoder is used to obtain a first fused feature based on the first spatial feature, the first frequency domain feature and the first texture feature extracted from the preprocessed wafer image to be inspected, and to obtain a latent feature based on the first fused feature. The multimodal classifier is used to obtain the probability distribution of each defect type based on the latent feature.

[0031] In one specific embodiment, step 3 may include: Step 3.1: Input the preprocessed wafer image to be inspected into the multimodal feature encoder to extract the first spatial feature, the first frequency domain feature and the first texture feature from the preprocessed wafer image to be inspected.

[0032] Step 3.11: Input the preprocessed wafer image to be inspected into the spatial feature extraction module of the multimodal feature encoder. The preprocessed wafer image to be inspected is processed by a three-level convolutional downsampling network to obtain downsampled features. The downsampled features are flattened and input into the first fully connected layer, and mapped into the first spatial features through the first fully connected layer.

[0033] Specifically, the preprocessed wafer image to be inspected is input into the spatial feature extraction module. In the spatial feature extraction module, the preprocessed wafer image to be inspected is first processed by a three-level convolutional downsampling network. The three-level convolutional downsampling network outputs downsampled features. The size of the downsampled features is, for example, 4×4×128. The downsampled features are flattened and then processed by the first fully connected layer to obtain the first spatial feature. The first spatial feature is a 512-dimensional feature vector. The macroscopic geometry and spatial distribution of defects are captured through the first spatial feature.

[0034] Optionally, the three-level convolutional downsampling network includes a first convolutional layer, a first activation function, a first downsampling layer, a second convolutional layer, a second activation function, a second downsampling layer, a third convolutional layer, a third activation function, and a third downsampling layer. The kernel size of the first, second, and third convolutional layers is 4×4, the stride is 2, the padding is 1, and the number of output channels is 32, 64, and 128, respectively. The first, second, and third activation functions are ReLU activation functions.

[0035] Step 3.12: Input the preprocessed wafer image to be inspected into the frequency domain feature extraction module of the multimodal feature encoder. The preprocessed wafer image to be inspected is processed by the fourth convolutional layer to obtain four frequency band features. The four frequency band features are respectively subjected to global average pooling to obtain four global average pooling features. The four global average pooling features are simultaneously input into two second fully connected layers to obtain four frequency band attention weights. Each frequency band attention weight is weighted with the corresponding frequency band feature to obtain four enhanced frequency band features. Each enhanced frequency band feature is subjected to adaptive pooling compression to obtain adaptive pooling compressed frequency band features. The four adaptive pooling compressed frequency band features are flattened into the first frequency domain features.

[0036] Specifically, the preprocessed wafer image to be inspected is input into the frequency domain feature extraction module. In the frequency domain feature extraction module, the preprocessed wafer image to be inspected is first decomposed by the fourth convolutional layer to obtain four frequency band features. Each frequency band feature is then subjected to global average pooling, resulting in four global average pooling features. These four global average pooling features are then processed by two second fully connected layers simultaneously, generating frequency band attention weights for each frequency band feature, resulting in four frequency band attention weights. Each frequency band attention weight is then weighted with its corresponding frequency band feature to enhance the frequency band feature, thus obtaining four enhanced frequency band features. Subsequently, each enhanced frequency band feature is subjected to 4×4 adaptive pooling compression, resulting in four adaptive pooling compressed frequency band features. Finally, the four adaptive pooling compressed frequency band features are flattened into a 64-dimensional first frequency domain feature, which captures the periodic patterns and frequency domain responses of defects.

[0037] Optionally, the fourth convolutional layer includes four 3×3 convolutional kernels.

[0038] Step 3.13: Input the preprocessed wafer image to be inspected into the texture feature extraction module of the multimodal feature encoder. The preprocessed wafer image to be inspected is processed by a three-level convolutional network to obtain convolutional features. The convolutional features are then processed by two layers of third fully connected layers to obtain the attention weights of each channel of the convolutional features. The attention weights of each channel are weighted with the convolutional features of the corresponding channel to obtain the weighted features of each channel. Each weighted feature is then compressed by adaptive pooling to obtain the weighted features after adaptive pooling. The weighted features after adaptive pooling of each channel are then flattened into the first texture feature.

[0039] Specifically, the preprocessed wafer image to be inspected is input into the texture feature extraction module. In the texture feature extraction module, the preprocessed wafer image to be inspected is first processed through a three-level convolutional network to extract texture features step by step, thus obtaining convolutional features. The convolutional features are input into two fully connected layers for nonlinear transformation and dimensionality reduction to learn the relationship between texture channels and output the attention weights of each channel of the convolutional features. Then, the attention weights of each channel are weighted with the convolutional features of the corresponding channel to obtain weighted features of each channel. This allows the key texture regions to be focused through the texture attention mechanism. Then, the weighted features of each channel are adaptively pooled and compressed to obtain 2*2 adaptively pooled and compressed weighted features of each channel. Finally, the adaptively pooled and compressed weighted features of each channel are flattened into 128-dimensional first texture features, which can be used to analyze the microstructure and roughness changes of the surface.

[0040] Optionally, the three-level convolutional network includes a fifth convolutional layer, a fourth activation function, a sixth convolutional layer, a fifth activation function, a seventh convolutional layer, and a sixth activation function. The kernel size of the fifth, sixth, and seventh convolutional layers is 3×3, and the number of channels is 16, 32, and 32, respectively. The fourth, fifth, and sixth activation functions are ReLU activation functions.

[0041] Step 3.2: Obtain the first fusion feature based on the first spatial feature, the first frequency domain feature, and the first texture feature.

[0042] Step 3.21: Project the first spatial feature, the first frequency domain feature, and the first texture feature onto a unified feature space of the same dimension through the fourth fully connected layer, the fifth fully connected layer, and the sixth fully connected layer, respectively, to obtain the second spatial feature, the second frequency domain feature, and the second texture feature of the same dimension.

[0043] Specifically, the 512-dimensional first spatial feature, the 64-dimensional first frequency domain feature, and the 128-dimensional first texture feature are projected onto a 256-dimensional unified feature space through the fourth, fifth, and sixth fully connected layers, respectively, to obtain the 256-dimensional second spatial feature, second frequency domain feature, and second texture feature.

[0044] Step 3.22: Stack the second spatial feature, the second frequency domain feature, and the second texture feature into a sequence containing three elements.

[0045] Specifically, the three 256-dimensional second spatial features, second frequency domain features, and second texture features are stacked into a sequence of 3 elements, each element representing a mode.

[0046] Step 3.23: Use the multi-head attention mechanism to enable cross-modal information interaction among the three elements in the sequence to obtain the third spatial feature, the third frequency domain feature, and the third texture feature.

[0047] Specifically, the multi-head attention mechanism allows all elements in the sequence to learn from each other. Each element calculates its correlation with itself and with the other two elements. Then, based on the calculated pairwise correlations, the features are weighted, thereby generating a new feature that incorporates information from other modalities. In this way, third spatial features, third frequency domain features, and third texture features can be obtained.

[0048] Step 3.24: Perform residual connections between the second spatial feature, the second frequency domain feature, and the second texture feature and the third spatial feature, the third frequency domain feature, and the third texture feature to obtain the fourth spatial feature, the fourth frequency domain feature, and the fourth texture feature.

[0049] Specifically, the second spatial feature and the third spatial feature are residually connected to obtain the fourth spatial feature, the second frequency domain feature and the third frequency domain feature are residually connected to obtain the fourth frequency domain feature, and the second texture feature and the third texture feature are residually connected to obtain the fourth texture feature.

[0050] Step 3.25: Concatenate the fourth spatial feature, the fourth frequency domain feature, and the fourth texture feature to obtain the concatenated feature.

[0051] Here, the concatenated features are 768-dimensional feature vectors.

[0052] Step 3.26: Input the spliced ​​features into the gated fusion network to obtain three modal weights.

[0053] Optionally, the gated fusion network includes a seventh fully connected layer, a seventh activation function, an eighth fully connected layer, and an eighth activation function, wherein the seventh activation function is a ReLU activation function and the eighth activation function is a softmax activation function.

[0054] Step 3.27: Perform a weighted summation of the three modal weights with the corresponding fourth spatial feature, fourth frequency domain feature, and fourth texture feature to obtain the first fused feature.

[0055] Here, the first fusion feature is a 256-dimensional feature vector.

[0056] Step 3.3: Map the first fused feature to the latent space to obtain the latent feature.

[0057] Specifically, the first fused features are mapped to a 64-dimensional latent space through the ninth fully connected layer to obtain 64-dimensional latent features.

[0058] Step 3.4: Input the latent features into the multimodal classifier to obtain the probability distribution of each defect type.

[0059] Step 3.41: Process the latent features through spatial branching, frequency domain branching, and texture branching respectively to obtain the fifth spatial feature, the fifth frequency domain feature, and the fifth texture feature.

[0060] Specifically, the multimodal classifier includes a spatial branch, a frequency domain branch, and a texture branch. The spatial branch, frequency domain branch, and texture branch are three independent branches. Latent features are transmitted to the spatial branch, frequency domain branch, and texture branch, respectively. The spatial branch outputs a 64-dimensional fifth spatial feature, the frequency domain branch outputs a 32-dimensional fifth frequency domain feature, and the texture branch outputs a 32-dimensional fifth texture feature.

[0061] Optionally, the spatial branch (128→64 dimensions), frequency domain branch (64→32 dimensions), and texture branch (64→32 dimensions) have the same structure, all including a tenth fully connected layer, a ninth activation function, a dropout layer, an eleventh fully connected layer, and a tenth activation function, where the ninth and tenth activation functions are both ReLU activation functions.

[0062] Step 3.42: Perform a weighted summation of the fifth spatial feature, the fifth frequency domain feature, and the fifth texture feature to obtain the second fused feature.

[0063] Here, the sum of the weights of the fifth spatial feature, the fifth frequency domain feature, and the fifth texture feature is 1, and the second fusion feature is a 128-dimensional feature vector.

[0064] Step 3.43: Obtain the probability distribution of each defect type based on the second fusion feature.

[0065] Specifically, the second fusion feature outputs the classification logic value of each defect type through the twelfth fully connected layer, thereby determining the probability distribution of each defect type.

[0066] Step 3.5: Select the defect type with the highest probability from the probability distribution of each defect type as the final defect classification result.

[0067] In an optional embodiment, a training method for a multimodal Wasserstein autoencoder is also provided, the training method comprising: S1. Obtain labeled training dataset and unlabeled training dataset. The labeled training dataset includes multiple preprocessed images of the wafer to be trained and their corresponding defect type labels. The unlabeled training dataset includes multiple preprocessed images of the wafer to be trained.

[0068] Specifically, wafer images and their corresponding defect type labels are loaded from a wafer defect data source (such as the WM811K dataset), and each wafer image is preprocessed to obtain a preprocessed wafer image. Then, based on the validity of the defect type labels, all the obtained preprocessed wafer images are divided into labeled datasets and unlabeled datasets according to whether they have defect type labels. The labeled datasets are further divided into labeled training datasets and validation datasets according to a preset ratio (such as 9:1), and the unlabeled datasets are used as unlabeled training datasets. Therefore, each frame of preprocessed wafer image in both the labeled training dataset and the unlabeled training dataset is used as a preprocessed wafer image to be trained.

[0069] Furthermore, the preprocessing steps for the wafer images are as follows: first, size normalization is performed to normalize the size to 32×32, followed by pixel value normalization to normalize the pixel values ​​to the range [-1, 1]. Additionally, random data augmentation can be applied to the wafer images used for training. Therefore, with a probability of 0.5, after size normalization, all training wafer images can be sequentially subjected to random horizontal flipping, random vertical flipping, random 90-degree rotation, and random brightness adjustment (brightness factor uniformly sampled within the range [0.8, 1.2]), before pixel value normalization is performed again.

[0070] S2. Divide the labeled training dataset and the unlabeled training dataset into two parts. N Each batch is sequentially input into a multimodal Wasserstein autoencoder. The Adam optimizer, combined with gradient pruning, is used to backpropagate and update the parameters of the multi-objective loss function, resulting in a trained multimodal Wasserstein autoencoder.

[0071] Specifically, labeled and unlabeled training datasets are combined and divided into multiple batches. Each batch includes several preprocessed wafer images from the labeled training dataset along with their corresponding defect type labels, and several preprocessed wafer images from the unlabeled training dataset. All batches are then sequentially input into a multimodal Wasserstein autoencoder. Using the Adam optimizer and gradient clipping techniques, backpropagation and parameter updates are performed on the multi-objective loss function until a well-trained multimodal Wasserstein autoencoder is obtained. During training, the accuracy of classification is continuously monitored on the validation set.

[0072] Furthermore, this embodiment adopts an early stopping strategy. The patience value of the early stopping strategy can be set to 10, that is, the training process is terminated when the accuracy of the validation set does not improve to a new highest value within 10 consecutive training cycles.

[0073] In this embodiment, the training process of the multimodal Wasserstein autoencoder requires the use of a feature reconstruction decoder. The feature reconstruction decoder is used to receive the latent features to be trained output by the multimodal feature encoder. The latent features to be trained are passed through the twelfth fully connected layer to obtain extended features. The extended features are then upsampled through three transposed convolutional layers to obtain the reconstructed image.

[0074] Specifically, after the wafer image to be trained is processed by the multimodal feature encoder, a 64-dimensional latent feature to be trained is obtained through the ninth fully connected layer. This latent feature to be trained is input into the feature reconstruction decoder, and the latent feature to be trained is expanded into a 128×4×4 extended feature through the twelfth fully connected layer. Then the extended feature is upsampled through three transposed convolutional layers to obtain a 32×32×1 reconstructed image.

[0075] Optionally, the three transposed convolutional layers include a first transposed convolutional layer, a ReLU activation function, a second transposed convolutional layer, a ReLU activation function, a third transposed convolutional layer, and a Tahn activation function. The kernel size of the first transposed convolutional layer, the second transposed convolutional layer, and the third transposed convolutional layer is 4×4, the stride is 2, the padding is 1, and the number of output channels is 64, 32, and 1, respectively.

[0076] In one specific embodiment, the multi-objective loss function consists of four parts that collectively guide the semi-supervised training of the model.

[0077] (1) Reconstruction Loss. The reconstruction loss uses the L1 loss function to measure the pixel-level difference between the reconstructed image output by the decoder and the original input image. The reconstruction loss is expressed as: ; in, To reconstruct the loss, The number of samples in each batch. For the first i The nth sample (i.e., the nth sample in the current batch) i (Image of the training wafer after frame preprocessing) For the first i A reconstructed image.

[0078] (2) Distribution Matching Loss. The distribution matching loss uses the maximum mean difference based on the kernel of the inverse multivariate quadratic function to constrain the consistency between the population distribution of the latent space and the standard normal prior. The distribution matching loss is expressed as: ; in, For distributed matching loss, For the first i One potential feature to be trained For the first j One potential feature to be trained The first sampled from the standard normal distribution i A random vector, The first sampled from the standard normal distribution j A random vector, It is an inverse multivariate quadratic kernel function.

[0079] (3) Classification Loss. The classification loss uses a weighted cross-entropy loss function, calculating weights based on inverse class frequencies to address the imbalance in the number of defective samples across different classes in the training data. The classification loss is expressed as: ; in, For classifying losses, This represents the total number of defect types. For the first c Weighting coefficients for each defect type For the first i The sample at the th c The true label (one-hot encoded) on each defect type, For the predicted first i The sample belongs to the first c The probability of each defect type.

[0080] (4) Modality consistency loss. Modality consistency loss is calculated by summing the variance and standard deviation of the encoder gating fusion weights and the classifier branch weights, ensuring that the two components maintain a consistent perception of modality importance. The modality consistency loss is expressed as: ; in, For modal consistency loss, The modal weights output by the gated fusion network of the multimodal feature encoder. The weights of each branch of the multimodal classifier are given. For variance, Standard deviation, This is the mean.

[0081] In this embodiment, the multi-objective loss function is expressed as: ; in, For multi-target loss, , , These are the preset weighting coefficients.

[0082] For the training wafer images in the unlabeled training dataset, when calculating the multi-objective loss, both the classification loss and the modality consistency loss in the multi-objective loss function are set to 0.

[0083] After training is complete, save the model weights that perform best on the validation set.

[0084] To verify the effectiveness and superiority of the multimodal feature fusion-based adversarial autoencoder-based wafer defect detection method provided in this invention for wafer defect classification, simulation and comparative experiments were conducted using the publicly available WM811K wafer defect dataset. This dataset contains various defect types and defect-free samples, totaling over 800,000 wafer images.

[0085] In the simulation, the weight adjustment factors in the multi-objective loss function were set as follows: reconstruction loss weight was 1.0, distribution matching loss weight was 15.0, classification loss weight was 0.8, and modality consistency loss weight was 0.1. The batch size was 128, the maximum training epochs were 100, the number of random experiments was 5, and the dropout rate was 0.5. Wafer images were uniformly scaled to 32×32 pixels, and pixel values ​​were normalized to the [-1, 1] interval.

[0086] The WM811K dataset contains 9 defect types, as shown in Table 1: Table 1 Defect Types

[0087] The WM811K dataset was divided into training, validation, and test sets in a 7:2:1 ratio. Only 10% of the samples in the training set were used as labeled data, while the remaining 90% were used as unlabeled data for training, to verify the semi-supervised learning effect of this invention.

[0088] Data augmentation strategies include: random rotation (0°, 90°, 180°, 270°), random horizontal flip (probability 0.5), random vertical flip (probability 0.5), and random brightness adjustment (brightness factor 0.8-1.2).

[0089] After training, the model's classification performance on the test set is as follows: Figure 2 As shown in Table 2, the overall classification accuracy is as follows: Table 2 Overall Classification Accuracy

[0090] The wafer defect detection method provided by this invention employs a modal consistency constraint mechanism to ensure consistency in the multimodal weight perception between the multimodal feature encoder and the multimodal classifier. It also proposes a dynamic weight adjustment method to address the problem of unbalanced wafer defect categories and calculates the loss function based on inverse weighting of category frequencies. Furthermore, it combines a semi-supervised training strategy with both labeled and unlabeled data to construct a multi-objective optimization framework comprising reconstruction loss, distribution matching loss, classification loss, and consistency loss. Therefore, the wafer defect detection method provided by this invention achieves good classification accuracy, a relatively stable convergence process, and reduces the requirement for labeled data. This wafer defect detection method helps improve the efficiency of quality inspection in semiconductor manufacturing processes and has a positive effect on the synergistic optimization of multimodal feature fusion and semi-supervised learning. The wafer defect detection method provided by this invention is not only suitable for lightweight network structure design in industrial inspection but also supports real-time defect classification and online model updates.

[0091] To address the defect classification problem in wafer manufacturing, traditional convolutional neural networks and standard autoencoders tend to overfit and find local optima. This invention, however, employs a multimodal Wasserstein autoencoder that demonstrates advantages across multiple performance metrics. The wafer defect detection method provided by this invention uses a multimodal Wasserstein autoencoder architecture and leverages the Wasserstein distance metric of the WAE to effectively solve the problems of unstable training and ambiguous feature representations in traditional generative models on wafer defect data. This results in learning clearer and more discriminative defect feature representations. Addressing the issues of scarce labeled data, imbalanced samples, and insufficient feature representation in wafer defect classification, this invention utilizes multimodal feature extraction, modality consistency constraints, and dynamic weight adjustment techniques to achieve efficient and accurate defect classification with limited labeled data. This significantly improves the accuracy and robustness of wafer defect classification, reduces reliance on labeled data, and enhances quality control in the semiconductor manufacturing process.

[0092] It should be noted that the terms "first," "second," etc., are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the above exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the present invention.

[0093] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Furthermore, those skilled in the art can combine and integrate the different embodiments or examples described in this specification.

[0094] Although the invention has been described herein in conjunction with various embodiments, those skilled in the art will understand and implement other variations of the disclosed embodiments by reviewing the accompanying drawings and the disclosure in carrying out the claimed invention. In the description of the invention, the word "comprising" does not exclude other components or steps, "a" or "an" does not exclude a plurality, and "a plurality" means two or more, unless otherwise explicitly specified. Furthermore, while different embodiments may describe certain measures, this does not mean that these measures cannot be combined to produce good results.

[0095] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. A wafer defect detection method based on multimodal feature fusion and adversarial autoencoder, characterized in that, The wafer defect detection method includes: Acquire an image of the wafer to be inspected; The image of the wafer to be inspected is preprocessed to obtain a preprocessed image of the wafer to be inspected; The preprocessed wafer image to be inspected is input into the trained multimodal Wasserstein autoencoder. The multimodal feature encoder and multimodal classifier of the trained multimodal Wasserstein autoencoder are used to perform forward propagation to obtain the probability distribution of each defect type, and the defect type corresponding to the highest probability is selected as the final defect classification result. The multimodal feature encoder is used to obtain a first fusion feature based on the first spatial feature, the first frequency domain feature, and the first texture feature extracted from the preprocessed wafer image to be inspected, and to obtain a latent feature based on the first fusion feature. The multimodal classifier is used to obtain the probability distribution of each defect type based on the latent feature.

2. The wafer defect detection method according to claim 1, characterized in that, The image of the wafer to be inspected is preprocessed to obtain a preprocessed image of the wafer to be inspected, including: The wafer image to be inspected is subjected to size normalization and pixel value normalization in sequence to obtain the preprocessed wafer image to be inspected.

3. The wafer defect detection method according to claim 1, characterized in that, The preprocessed wafer image to be inspected is input into a trained multimodal Wasserstein autoencoder. The multimodal feature encoder and multimodal classifier of the trained multimodal Wasserstein autoencoder are then forward-propagated to obtain the probability distribution of each defect type. The defect type with the highest probability is selected as the final defect classification result, including: The preprocessed wafer image to be inspected is input into the multimodal feature encoder to extract the first spatial feature, the first frequency domain feature, and the first texture feature from the preprocessed wafer image to be inspected. The first fusion feature is obtained based on the first spatial feature, the first frequency domain feature, and the first texture feature; The first fused feature is mapped to the latent space to obtain the latent feature; The latent features are input into the multimodal classifier to obtain the probability distribution of each defect type; The defect type with the highest probability from the probability distribution of each defect type is selected as the final defect classification result.

4. The wafer defect detection method according to claim 3, characterized in that, The preprocessed wafer image to be inspected is input into the multimodal feature encoder to extract the first spatial feature, the first frequency domain feature, and the first texture feature from the preprocessed wafer image to be inspected, including: The preprocessed wafer image to be inspected is input into the spatial feature extraction module of the multimodal feature encoder. The preprocessed wafer image to be inspected is processed by a three-level convolutional downsampling network to obtain downsampled features. The downsampled features are flattened and input into the first fully connected layer, and mapped to the first spatial features by the first fully connected layer. The three-level convolutional downsampling network includes a first convolutional layer, a first activation function, a first downsampling layer, a second convolutional layer, a second activation function, a second downsampling layer, a third convolutional layer, a third activation function, and a third downsampling layer. The preprocessed wafer image to be detected is input into the frequency domain feature extraction module of the multimodal feature encoder. The preprocessed wafer image to be detected is passed through the fourth convolutional layer to obtain four frequency band features. The four frequency band features are respectively subjected to global average pooling to obtain four global average pooling features. The four global average pooling features are simultaneously input into two second fully connected layers to obtain four frequency band attention weights. Each frequency band attention weight is weighted with the corresponding frequency band feature to obtain four enhanced frequency band features. Each enhanced frequency band feature is subjected to adaptive pooling compression to obtain adaptive pooling compressed frequency band features. The four adaptive pooling compressed frequency band features are flattened into the first frequency domain features. The preprocessed wafer image to be detected is input into the texture feature extraction module of the multimodal feature encoder. The preprocessed wafer image to be detected is processed by a three-level convolutional network to obtain convolutional features. The convolutional features are passed through two third fully connected layers to obtain attention weights for each channel of the convolutional features. The attention weights of each channel are weighted with the convolutional features of the corresponding channel to obtain weighted features for each channel. Each weighted feature is compressed by adaptive pooling to obtain compressed weighted features. The compressed weighted features of each channel are flattened to form the first texture feature. The three-level convolutional network includes a fifth convolutional layer, a fourth activation function, a sixth convolutional layer, a fifth activation function, a seventh convolutional layer, and a sixth activation function.

5. The wafer defect detection method according to claim 3, characterized in that, The first fused feature is obtained based on the first spatial feature, the first frequency domain feature, and the first texture feature, including: The first spatial feature, the first frequency domain feature, and the first texture feature are projected onto a unified feature space of the same dimension through the fourth fully connected layer, the fifth fully connected layer, and the sixth fully connected layer, respectively, to obtain the second spatial feature, the second frequency domain feature, and the second texture feature of the same dimension. The second spatial feature, the second frequency domain feature, and the second texture feature are stacked into a sequence consisting of three elements; By utilizing a multi-head attention mechanism, three elements in the sequence are made to interact across modalities to obtain a third spatial feature, a third frequency domain feature, and a third texture feature. The second spatial feature, the second frequency domain feature, and the second texture feature are respectively coupled with the third spatial feature, the third frequency domain feature, and the third texture feature to obtain the fourth spatial feature, the fourth frequency domain feature, and the fourth texture feature; The fourth spatial feature, the fourth frequency domain feature, and the fourth texture feature are concatenated to obtain the concatenated feature; The concatenated features are input into a gated fusion network to obtain three modal weights. The first fused feature is obtained by weighting and summing the three modal weights with the corresponding fourth spatial feature, fourth frequency domain feature, and fourth texture feature.

6. The wafer defect detection method according to claim 3, characterized in that, The latent features are input into the multimodal classifier to obtain the probability distribution of each defect type, including: The potential features are processed through spatial branching, frequency domain branching, and texture branching respectively to obtain the fifth spatial feature, the fifth frequency domain feature, and the fifth texture feature; The second fused feature is obtained by weighted summation of the fifth spatial feature, the fifth frequency domain feature, and the fifth texture feature; The probability distribution of each defect type is obtained based on the second fusion feature.

7. The wafer defect detection method according to claim 3, characterized in that, The multimodal Wasserstein autoencoder also Includes a feature reconstruction decoder; The training method for the multimodal Wasserstein autoencoder includes: Obtain a labeled training dataset and an unlabeled training dataset. The labeled training dataset includes multiple preprocessed images of the wafer to be trained and corresponding defect type labels. The unlabeled training dataset includes multiple preprocessed images of the wafer to be trained. The labeled training dataset and the unlabeled training dataset are divided into multiple batches and sequentially input into the multimodal Wasserstein autoencoder. The Adam optimizer is used, combined with gradient pruning technology, to perform backpropagation and parameter update on the multi-objective loss function, resulting in a trained multimodal Wasserstein autoencoder.

8. The wafer defect detection method according to claim 7, characterized in that, The feature reconstruction decoder is used to receive the latent features to be trained output by the multimodal feature encoder. The latent features to be trained are passed through the thirteenth fully connected layer to obtain extended features. The extended features are then upsampled through three transposed convolutional layers to obtain the reconstructed image.

9. The wafer defect detection method according to claim 8, characterized in that, The multi-objective loss function is expressed as: ； in, For multi-target loss, To reconstruct the loss, For distributed matching loss, For classifying losses, For modal consistency loss, , , These are preset weighting coefficients; For the training wafer images in the unlabeled training dataset, the classification loss and modality consistency loss in the multi-objective loss function are both set to 0.

10. The wafer defect detection method according to claim 9, characterized in that, The reconstruction loss, the distribution matching loss, the classification loss, and the mode consistency loss are respectively expressed as: ； ； ； ； in, The number of samples in each batch. This represents the total number of defect types. For the first i One sample, For the first i A reconstructed image, For the first i One potential feature to be trained For the first j One potential feature to be trained The first sampled from the standard normal distribution i A random vector, The first sampled from the standard normal distribution j A random vector, It is an inverse multivariate quadratic kernel function. For the first c Weighting coefficients for each defect type For the first i The sample at the th c The true label on each defect type For the predicted first i The sample belongs to the first c The probability of each defect type The modal weights output by the gated fusion network of the multimodal feature encoder. The weights of each branch of the multimodal classifier are given. For variance, Standard deviation This is the mean.

Images