Wafer map generative data augmentation method based on denoising diffusion probability model

Abstract

This invention proposes a wafer image generative data augmentation method based on a denoising diffusion probability model to supplement the original wafer image dataset and address the problem of poor model generalization ability caused by imbalanced sample distribution in wafer image defect recognition tasks. The method includes: acquiring the original wafer image defect dataset and performing uniform size standardization preprocessing on the dataset; constructing a denoising diffusion probability model adapted to the wafer image features, learning the distribution characteristics of real wafer images through a diffusion process of progressively adding noise and then reverse denoising; training the denoising diffusion probability model using the preprocessed wafer image dataset, enabling the model to generate high-fidelity, diverse wafer images containing real defect features; and fusing the generated wafer images with the original dataset to form an expanded balanced wafer image dataset. The wafer image samples generated by this method can effectively supplement the original data distribution, significantly improve the training effect of subsequent wafer image defect recognition models, and provide reliable data support for the defect detection stage in the wafer manufacturing process.

Description

Technical Field

[0001] This invention relates to the field of wafer defect detection in semiconductor manufacturing processes, and more specifically to a wafer image generative data augmentation method based on a denoising diffusion probability model. Background Technology

[0002] Semiconductor wafer manufacturing is a highly precise and complex production process. Various defects on the wafer surface directly affect chip performance and finished product yield. Therefore, wafer image defect identification is a core link in semiconductor production line quality control, defect tracing, and process optimization. It plays an irreplaceable and important role in ensuring production stability and improving chip manufacturing yield.

[0003] In real-world semiconductor manufacturing scenarios, production lines typically exhibit high stability and yield, resulting in a severe imbalance in wafer defect data distribution: normal samples and majority class defect samples dominate, while minority class defect samples occur with extremely low frequency and are extremely scarce. This data imbalance leads to insufficient learning of minority class defect features by the trained defect recognition model, significantly reducing the accuracy of minority class defect recognition. This directly impacts the reliability of defect tracing and quality control on the production line, hindering the practical application of defect detection technology in the semiconductor manufacturing field.

[0004] However, in real-world industrial scenarios, the manual annotation and collection of wafer image defect samples is complex and costly, making it difficult to quickly construct large-scale labeled datasets. Traditional data augmentation methods generate samples solely through geometric transformations, resulting in high feature homogeneity and limited improvement in the generalization performance of defect recognition models. In contrast, diffusion models can deeply fit the spatial structure and defect texture of real wafer images through a progressive forward noise addition and reverse denoising process, generating highly diverse wafer image defect samples that closely resemble real-world production scenarios. This effectively avoids the feature homogeneity problem of traditional augmentation methods. By supplementing scarce defect data with the generated samples, the training effect and generalization ability of defect recognition models are significantly improved, providing a new technical approach to solving the problem of data imbalance in wafer image defect recognition tasks. Summary of the Invention

[0005] The purpose of this invention is to provide a wafer image generative data augmentation method based on a denoising diffusion probability model. By supplementing the original imbalanced dataset with generative data augmentation, the accuracy and generalization ability of existing wafer image defect recognition models can be improved.

[0006] To achieve the above objectives, the present invention provides a wafer pattern generative data augmentation method, comprising:

[0007] Step 1. Preprocess the original wafer image defect dataset by using bilinear interpolation to unify the wafer image size;

[0008] Step 2. Construct a deep learning network model with a U-Net structure;

[0009] Step 3. Training phase: Gaussian noise is added to the preprocessed wafer image using a noise scheduler to obtain a noisy wafer image. The noisy wafer image and the corresponding time step are then input into the constructed deep learning network model to train the model.

[0010] Step 4. Generation stage: Randomly generate Gaussian noise, input it into the trained deep learning network model, predict the noise at the current time, and iteratively remove noise to restore and generate the wafer image;

[0011] Step 5. Merge the wafer image generated by the trained denoising diffusion model with the original dataset to generate a balanced wafer image dataset, thus completing the wafer image data augmentation.

[0012] In step 1, the preprocessing of the wafer image involves converting the wafer image data into an image with a size of 32×32 and 1 pixel channel, and normalizing the pixel values ​​of the wafer image to the range of [-1,1] through a normalization operation.

[0013] In step 2, the constructed deep learning network model is a denoising diffusion model adapted to wafer image features. Its core feature extraction network is a customized U-net network, which includes three parts: a downsampling encoding module, a bottleneck feature aggregation module, and an upsampling decoding module.

[0014] The downsampling encoding module is used to extract spatial texture and defect features of the wafer image layer by layer. It contains one convolutional layer and three downsampling blocks. Each downsampling block consists of multiple residual blocks, multi-head attention blocks, and downsampling convolutional layers. Convolutional downsampling operations are performed on the downsampling convolutional layers of the first and second downsampling blocks to halve the feature map size. The wafer image dimension input to the downsampling module is [1, 32, 32], and the output feature map dimension after downsampling by the downsampling encoding module is [256, 8, 8].

[0015] The first downsampling input feature map has dimensions [64, 32, 32], and the output feature map has dimensions [64, 16, 16].

[0016] The second downsampling input feature map has dimensions [128, 16, 16], and the output feature map has dimensions [128, 8, 8].

[0017] The bottleneck feature aggregation module is used to fuse deep global features of the wafer map. It includes two bottleneck blocks and a hole space pyramid pooling module, without downsampling or upsampling operations. Each bottleneck block consists of multiple residual blocks and multi-head attention blocks. The hole space pyramid module extracts multi-scale contextual information of the input feature map through 5 parallel branches and finally fuses the output features of a unified scale. The feature map dimension of the input bottleneck feature aggregation module is [256, 8, 8], and the output feature map dimension is [128, 8, 8].

[0018] The upsampling decoding module is used to recover the spatial dimensions of the wafer image layer by layer. It includes three upsampling blocks, one normalization layer, one SiLU layer, and one convolutional layer. Each upsampling block consists of an upsampling convolutional layer, multiple residual blocks, and a multi-head attention block. Transposed convolutional upsampling operations are performed on the upsampling convolutional layers of the second and third upsampling blocks to double the feature map size. After each upsampling convolutional layer, a skip connection is performed to fuse the feature map of the corresponding stage of the downsampling encoding module. The feature map dimension of the input upsampling module is [128, 8, 8]. After upsampling by the upsampling decoding module, the output feature map dimension is [1, 32, 32], which is consistent with the original input wafer image dimension. The output is the noise prediction result of the noisy wafer image.

[0019] The first upsampling input feature map has dimensions [64, 8, 8], and the output feature map has dimensions [64, 16, 16].

[0020] The second upsampling input feature map has dimensions [32, 16, 16], and the output feature map has dimensions [32, 32, 32].

[0021] Furthermore, the multi-layer residual blocks contained in the downsampling encoding module, bottleneck feature aggregation module, and upsampling decoding module need to be integrated with temporal embedding. This integrates the time-step information of the diffusion process throughout the entire wafer image feature extraction process, enabling the network to capture the variation patterns of defect features in the wafer image under different noise levels. The specific implementation method is as follows:

[0022] Temporal embedding is performed by nonlinearly transforming the initial temporal embedding through a SiLU layer, followed by dimensional mapping through a linear layer. Multiple residual blocks sequentially perform grouping normalization, SiLU activation function transformation, and convolutional layer feature extraction on the input wafer image feature map. Finally, the dimension-mapped temporal embedding and the feature map after feature extraction are added channel-by-channel to achieve the fusion of temporal dimension information and wafer image spatial features. The fused feature map is then subjected to grouping normalization, SiLU activation function transformation, and convolutional layer feature transformation before being residually connected to the original input features of the multiple residual blocks, completing the temporal embedding integration and feature transformation of a single multiple residual block.

[0023] The temporal embedding is obtained by combining sine and cosine positional encoding with a two-layer linear mapping. Its specific implementation is as follows:

[0024] Based on the preset embedding dimension and diffusion time step, the current diffusion time step is initially encoded using sine and cosine positional encoding to generate a fixed-dimensional time feature vector. The obtained time feature vector is then sequentially input into the first linear transformation layer and the SiLU activation function to complete the first nonlinear mapping. The mapped features are then input into the second linear transformation layer to finally output the time embedding. Attached Figure Description

[0025] Figure 1 This is a schematic diagram illustrating the wafer pattern generation data enhancement method of the present invention;

[0026] Figure 2 This is a structural diagram of the core feature extraction network model constructed in this invention;

[0027] Figure 3 It is the void space pyramid pooling module in the model constructed in this invention;

[0028] Figure 4 It is the multi-layer residual block structure in the model constructed in this invention;

[0029] Figure 5 This is the time embedding calculation process in the model constructed in this invention. Detailed Implementation

[0030] The present invention will be further described below with reference to the accompanying drawings.

[0031] This invention employs a wafer image generative data augmentation method based on a denoising diffusion probability model, which augments the original imbalanced dataset through generative methods. The specific process is attached. Figure 1 As shown:

[0032] Step S101: Preprocess the original wafer image dataset, adjust the wafer images of different sizes to 32×32 with 1 pixel channel through bilinear interpolation, and perform normalization processing to normalize the pixel values ​​of the wafer images to the range of [-1,1].

[0033] Step S102: Build a denoising diffusion probability model adapted to wafer image features. Its core feature extraction network is a customized U-Net network that includes a downsampling encoding module, a bottleneck feature aggregation module, and an upsampling decoding module.

[0034] Step S103: Gaussian noise is added to the preprocessed wafer image according to the time step by the noise scheduler to obtain the noisy wafer image. The noisy wafer image and the corresponding time step are input into the constructed network model to train the model.

[0035] Step S104: Randomly generate Gaussian noise of the same size as the wafer image, input it into the network model trained in step S103, predict the noise at the current time, and restore the wafer image by iteratively removing the noise.

[0036] Step S105: Merge the wafer map generated by the model with the original wafer map dataset to generate a wafer map dataset with a balanced number of defects in each category, thus completing the wafer map data augmentation.

[0037] The network model proposed in this invention is attached. Figure 2 As shown, it consists of three parts: a downsampling encoding module, a bottleneck feature aggregation module, and an upsampling decoding module.

[0038] The main function of the downsampling encoding module is to extract spatial texture and defect features from the wafer image layer by layer. It contains one convolutional layer and three downsampling blocks. Each downsampling block consists of multiple residual blocks, multi-head attention blocks, and downsampling convolutional layers. The downsampling encoding module performs convolutional downsampling operations in the downsampling convolutional layers of the first and second downsampling blocks, respectively, to halve the feature map size. The first downsampling input feature map dimension is [64, 32, 32], and the output feature map dimension is [64, 16, 16]. The second downsampling input feature map dimension is [128, 16, 16], and the output feature map dimension is [128, 8, 8].

[0039] The main function of the bottleneck feature aggregation module is to fuse deep global features from the wafer map. It includes two bottleneck blocks and a hole space pyramid pooling module, with no downsampling or upsampling operations. Each bottleneck block consists of multiple layers of residual blocks and multi-head attention blocks. The hole space pyramid pooling module is attached. Figure 3 As shown, this module extracts multi-scale contextual information from the input feature map through one convolutional branch, three dilated convolutional branches, and one global average pooling branch. Then, it concatenates and fuses the features from the five branches along the channel dimension, which serves as the output of the dilated spatial pyramid pooling module. The feature map dimension of the input bottleneck feature aggregation module is [256, 8, 8], and the output feature map dimension is [128, 8, 8].

[0040] The main function of the upsampling decoding module is to recover the spatial dimensions of the wafer image layer by layer. It includes three upsampling blocks, one normalization layer, one SiLU layer, and one convolutional layer. Each upsampling block consists of an upsampling convolutional layer, multiple residual blocks, and a multi-head attention block. The upsampling decoding module performs transposed convolutional upsampling operations in the upsampling convolutional layers of the second and third upsampling blocks, doubling the feature map size. After each upsampling convolutional layer, it performs skip connections and fusion with the feature maps of the corresponding stage of the downsampling encoding module. The feature map dimension of the input upsampling module is [128, 8, 8]. After upsampling by the upsampling decoding module, the output feature map dimension is [1, 32, 32], which is consistent with the original input wafer image dimension. The output is the noise prediction result of the noisy wafer image. The first upsampling input feature map dimension is [64, 8, 8], and the output feature map dimension is [64, 16, 16]. The second upsampling input feature map has dimensions [32, 16, 16], and the output feature map has dimensions [32, 32, 32].

[0041] In this invention, the multi-layer residual blocks included in the downsampling encoding module, bottleneck feature aggregation module, and upsampling decoding module need to be integrated with temporal embedding. This integrates the time-step information of the diffusion process throughout the entire wafer image feature extraction process, enabling the network to capture the variation patterns of defect features in the wafer image under different noise levels. The specific implementation method is attached. Figure 4 As shown:

[0042] Temporal embedding involves a nonlinear transformation of the initial temporal embedding using a SiLU layer, followed by dimensional mapping via a linear layer. Multiple residual blocks sequentially perform grouping normalization, SiLU activation function transformation, and convolutional layer feature extraction on the input wafer image feature map. Finally, the dimension-mapped temporal embedding and the feature map after feature extraction are added channel-by-channel to fuse temporal dimension information with wafer image spatial features. The fused feature map undergoes further grouping normalization, SiLU activation function transformation, and convolutional layer feature transformation before being residually connected to the original input features of the multiple residual blocks, completing the temporal embedding integration and feature transformation of a single multiple residual block.

[0043] In this invention, the time embedding is obtained by combining sine and cosine position encoding with a two-layer linear mapping, and its specific implementation is shown in the appendix. Figure 5 As shown:

[0044] Based on the preset embedding dimension and diffusion time step, sine and cosine positional encoding is used to initially encode the current diffusion time step, generating a fixed-dimensional temporal feature vector. The obtained temporal feature vector is then sequentially input into the first linear transformation layer and the SiLU activation function to complete the first nonlinear mapping. The mapped features are then input into the second linear transformation layer, finally outputting the temporal embedding.

Claims

1. A wafer map generative data augmentation method based on a denoising diffusion probability model, characterized in that, Includes the following steps: Step 1. Preprocess the original wafer image defect dataset by using bilinear interpolation to unify the wafer image size; Step 2. Construct a deep learning network model with a U-Net structure; Step 3. Training phase: Gaussian noise is added to the preprocessed wafer image using a noise scheduler to obtain a noisy wafer image. The noisy wafer image and the corresponding time step are then input into the constructed deep learning network model to train the model. Step 4. Generation stage: Randomly generate Gaussian noise, input it into the trained deep learning network model, predict the noise at the current time, and iteratively remove noise to restore and generate the wafer image; Step 5. Merge the wafer image generated by the trained denoising diffusion model with the original dataset to generate a balanced wafer image dataset, thus completing the wafer image data augmentation.

2. The wafer image generative data augmentation method based on a denoising diffusion probability model as described in claim 1, characterized in that, The preprocessing of the wafer image involves converting the wafer image data into an image with a size of 32×32 and 1 pixel channel, and normalizing the pixel values ​​of the wafer image to the range of [-1,1] through a normalization operation.

3. The wafer image generative data augmentation method based on a denoising diffusion model as described in claim 1, characterized in that, In the constructed denoising diffusion model adapted to wafer image features, the core feature extraction network is a customized Unet network, which consists of a downsampling encoding module, a bottleneck feature aggregation module, and an upsampling decoding module. 3-1. The downsampling encoding module is used to extract the spatial texture and defect features of the wafer image layer by layer. It includes one convolutional layer and three downsampling blocks. Each downsampling block consists of multiple residual blocks, multi-head attention blocks, and downsampling convolutional layers. Convolutional downsampling operations are performed on the downsampling convolutional layers of the first and second downsampling blocks to halve the feature map size. The wafer image dimension input to the downsampling module is [1, 32, 32], and the output feature map dimension after downsampling by the downsampling encoding module is [256, 8, 8]. 3-2. The bottleneck feature aggregation module is used to fuse deep global features of the wafer map. It includes two bottleneck blocks and a hole space pyramid pooling module, without downsampling or upsampling operations. Each bottleneck block consists of multiple residual blocks and multi-head attention blocks. The hole space pyramid module extracts multi-scale contextual information of the input feature map through 5 parallel branches, and finally fuses and outputs features of a unified scale. The feature map dimension of the input bottleneck feature aggregation module is [256, 8, 8], and the output feature map dimension is [128, 8, 8]. 3-3. The upsampling decoding module is used to recover the spatial dimension of the wafer image layer by layer. It includes 3 upsampling blocks, 1 normalization layer, 1 SiLU layer and 1 convolutional layer. Each upsampling block consists of an upsampling convolutional layer, multiple residual blocks and multi-head attention blocks. Transposed convolutional upsampling operations are performed on the upsampling convolutional layers of the second and third upsampling blocks to double the feature map size. After each upsampling convolutional layer, the feature map is fused with the feature map of the corresponding stage of the downsampling encoding module. The feature map dimension of the input upsampling module is [128, 8, 8]. After upsampling by the upsampling decoding module, the output feature map dimension is [1, 32, 32], which is consistent with the original input wafer image dimension. The output is the noise prediction result of the noisy wafer image.

4. The wafer pattern generating data augmentation method as described in claim 3, characterized in that, The aforementioned multi-layer residual block needs to integrate temporal embedding, fusing time-step information of the diffusion process throughout the entire wafer image feature extraction process, enabling the network to capture the variation patterns of defect features in the wafer image under different noise levels. The specific implementation method is as follows: 4-1. Temporal embedding first undergoes a nonlinear transformation of the initial temporal embedding through a SiLU layer, followed by dimensional mapping through a linear layer; 4-2. The multi-layer residual block performs grouping normalization, SiLU activation function transformation, and convolutional layer feature extraction operations sequentially on the input wafer image feature map; 4-3. Perform a channel-by-channel addition operation on the temporal embedding after dimension mapping in step 4-1 and the feature map after processing in step 4-2 to achieve the fusion of temporal dimension information and wafer image spatial features; 4-4. The fused feature maps are further processed by grouping normalization, SiLU activation function transformation, and convolutional layer feature transformation. Then, residual connections are performed with the original input features of the multi-layer residual blocks to complete the temporal embedding integration and feature transformation of a single multi-layer residual block.

5. The wafer pattern generating data augmentation method as described in claim 4, characterized in that, The time embedding is obtained by combining sine and cosine position coding with a two-layer linear mapping. The specific steps are as follows: 5-1. Based on the preset embedding dimension and diffusion time step, the current diffusion time step is initially encoded using sine and cosine positional encoding to generate a fixed-dimensional time feature vector; 5-2. The obtained time feature vector is sequentially input into the first linear transformation layer and the SiLU activation function to complete the first nonlinear mapping; 5-3. Continue to input the mapped features into the second linear transformation layer, and finally output the temporal embedding.

Images