An image segmentation method with adaptive feature enhancement for chip wafer defects

Abstract

This invention discloses an image segmentation method with adaptive feature enhancement for chip wafer defects, belonging to the field of data processing technology. The method includes: acquiring a historical chip wafer production image database, constructing a multimodal dataset of chip wafer production defects, training a dual-branch encoder for defects, and generating semantic vectors and background structure vectors for chip wafer defects; training a multimodal segmentation network for chip wafer defects based on the semantic vectors and background structure vectors to generate multi-defect label segmentation vectors for chip wafer production images; extracting a feature pool of chip wafer defects from the encoder end of the multimodal segmentation network, performing association matching and filtering on real-time chip wafer production images, and substituting this feature into the multimodal segmentation network to generate multi-defect label segmentation vectors for real-time chip wafer production images. The beneficial effects are: improving the accuracy, robustness, and generalization ability of chip wafer defect segmentation to unknown scenarios.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, specifically to an image segmentation method with adaptive feature enhancement for chip wafer defects. Background Technology

[0002] Existing image segmentation methods often focus on improving model performance during the training phase through static data augmentation or single feature space transformation, lacking a systematic decoupling of defect semantics and background environment. This results in limited diversity and insufficient realism of the generated data. At the same time, these methods typically fix the model parameters, making it impossible to dynamically refer to historical prior knowledge for online adaptive reasoning when deployed in complex and ever-changing real-time production scenarios. Consequently, they have weak generalization ability for unknown defect morphologies and new background environments, making it difficult to meet the needs of high-precision industrial quality inspection. Summary of the Invention

[0003] To address the aforementioned technical problems, an image segmentation method with adaptive feature enhancement for chip wafer defects is provided. This technical solution solves the problems mentioned above.

[0004] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0005] An image segmentation method with adaptive feature enhancement for chip wafer defects includes:

[0006] S1. Obtain a database of historical chip wafer production images, filter historical chip wafer production images according to known defect categories in chip wafer production, construct a multimodal dataset of chip wafer production defects, train a defect dual-branch encoder, and generate chip wafer defect semantic vectors and background structure vectors.

[0007] S2. Based on the semantic vector of chip wafer defects and the background structure vector, a controllable hybrid chip wafer defect generator is constructed to generate a chip wafer defect superposition state sample dataset, train a chip wafer defect multimodal segmentation network, and generate multi-defect label segmentation vectors for chip wafer production images.

[0008] S3. Based on the encoder end of the chip wafer defect multimodal segmentation network, extract the global feature vector of multi-defect labels of chip wafer production images, construct the chip wafer defect source domain feature pool, perform association matching and filtering on real-time chip wafer production images, obtain the real-time chip wafer production image matching defect source domain map vector, substitute it into the chip wafer defect multimodal segmentation network, and generate the multi-defect label segmentation vector of real-time chip wafer production images.

[0009] Preferably, step S1 specifically includes:

[0010] Based on a historical chip wafer production image database, preprocessing is performed according to image normalization and size standardization to obtain the associated metadata of historical chip wafer production images;

[0011] Based on the metadata associated with historical chip wafer production images, substitute it into One-Hot encoding to obtain the metadata vector associated with historical chip wafer production images;

[0012] Based on the SAM arbitrary object segmentation model, the known defect categories in chip wafer production are taken as input, a cue encoder is constructed with the text description of the known defect categories, a mask encoder is constructed with the spatial information of the known defect categories in the image, a visual backbone network for known defect categories in chip wafer production is pre-trained, and a label vector for known defect categories in chip wafer production is generated.

[0013] UMAP nonlinear dimensionality reduction involves dimensionality reduction of the known defect category label vector and the associated element vector of historical chip wafer production images. Substituting these vectors into a binary scatter plot yields the scatter distribution of the dimensionality-reduced vector of the known defect category label vector and the scatter distribution of the dimensionality-reduced vector of the associated element vector of historical chip wafer production images.

[0014] Based on DBSCAN density clustering, the K-distance maps of the scatter distribution of dimensionality-reduced vectors of known defect category labels in chip wafer manufacturing and the scatter distribution of dimensionality-reduced vectors of associated elements in historical chip wafer manufacturing images are calculated. The minimum distance and number of neighbors between each defect category cluster label and the metadata cluster label are determined for clustering. This yields the defect category cluster to which the known defect category vectors in chip wafer manufacturing belong, and the metadata cluster label to which the associated element vectors in historical chip wafer manufacturing images belong.

[0015] Based on the chi-square test, the defect category clusters to which the known defect category vectors of chip wafer production belong are used as rows, and the metadata cluster labels to which the attached element vectors of historical chip wafer production images belong are used as tags. The independence probabilities of row clusters and column clusters are calculated to determine the association between the known defect categories of chip wafer production and the attached element vectors of historical chip wafer production images, thus constructing a multimodal dataset of chip wafer production defects.

[0016] Preferably, step S1 further includes:

[0017] Based on the ResNet34 convolutional neural network, the global average pooling layer and fully connected layer are removed to construct a backbone network for extracting defect contribution features. The multimodal dataset of chip wafer production defects is used as the input of the convolutional layer to generate low-level feature maps of each modality of chip wafer production defects.

[0018] The backbone is extracted based on the defect contribution features. A defect semantic branch is constructed using an MLP multilayer perceptron, and a background structure branch is constructed using a U-Net network. A defect dual-branch encoder is constructed. The low-level feature maps of each modality of chip wafer production defects are used as input to extract the defect information and the probability of each pixel value position as background in the low-level feature maps of each modality of chip wafer production defects. The intermediate feature maps and background probability maps of each modality of chip wafer production defects are obtained.

[0019] Based on the intermediate feature maps of various modalities of chip wafer manufacturing defects, the defects are substituted into the defect segmentation decoder. The global average pooling layer is used to extract the binary defect masks of various modalities of chip wafer manufacturing defects. The attention weights of the binary defect masks of each modality are obtained by downsampling to the same spatial size as the intermediate feature maps of various modalities of chip wafer manufacturing defects. Weighted global average pooling is then performed on the intermediate feature maps of various modalities of chip wafer manufacturing defects to generate the semantic vector of chip wafer defects.

[0020] Based on the background probability maps of various modalities of chip wafer manufacturing defects, a class-independent segmentation decoder is introduced. Using a global average pooling layer, the modal background probability maps are binarized into background region masks. The high-level semantic feature maps of chip wafer defects are extracted from the defect contribution feature extraction backbone network. The modal background probability maps are then binarized into background region masks and downsampled to the same size as the high-level feature maps. The attention weights of the background region masks are then weighted and global average pooled to generate the background structure vector of chip wafer defects.

[0021] Preferably, step S2 specifically includes:

[0022] Based on the CGAN conditional generative adversarial network architecture, a controllable hybrid chip wafer defect generator is established according to SPADE spatial adaptive normalization. The chip wafer defect background structure vector is used as the generated image basis, and the chip wafer defect semantic vector is used as the defect type generation condition. According to the image basis and defect type generation conditions, a defect heatmap and random defect center positions are generated. The distance of each pixel in the defect heatmap to the random defect center position is calculated by Euclidean distance. The distance is substituted into a two-dimensional Gaussian function to obtain the intensity value of each random defect center position in the defect heatmap, thus obtaining the chip wafer production image defect spatial location heatmap.

[0023] The spatial location heatmap of defects in chip wafer manufacturing images and the multimodal dataset of chip wafer manufacturing defects are substituted into the multi-defect task discriminator. The least squares loss is used as the adversarial loss function of the multimodal task discriminator to output the probability that the defects in the chip wafer manufacturing images are the true data distribution. The multi-class cross-entropy loss is used as the auxiliary segmentation function to verify and predict the image segmentation mask in the input multimodal dataset of chip wafer manufacturing defects. This mask serves as the constraint condition for the generated image segmentation structure of the controllable hybrid chip wafer defect generator, generating chip wafer defect superposition state image-mask pairs and constructing a chip wafer defect superposition state sample dataset.

[0024] Preferably, step S2 further includes:

[0025] Based on the chip wafer defect superposition state sample dataset and the chip wafer production defect multimodal dataset, an original training set for chip wafer defects was constructed.

[0026] Based on the HRNet high-resolution network, a backbone network for multimodal segmentation of chip wafer defects is established.

[0027] The original training set of chip wafer defects is substituted into the backbone network of the chip wafer defect multimodal segmentation network. The chip wafer defect category and defect background of the last convolutional layer of the network are input into the Sigmoid activation function, and the chip wafer defect multi-class probability tensor is output.

[0028] Based on binary cross-entropy, a pixel-wise multi-task binary cross-entropy loss is established. The loss function between the mask pair of the chip wafer defect multi-class probability tensor output by the backbone network of the chip wafer defect multi-modal segmentation network and the chip wafer production defect image mask in the chip wafer production defect multi-modal dataset is verified. The pixel-wise multi-task binary cross-entropy loss is minimized by gradient descent to obtain the multi-defect label segmentation vector of the chip wafer production image.

[0029] Preferably, step S3 specifically includes:

[0030] Based on the encoder end of the chip wafer defect multimodal segmentation network, the spatial and semantic information feature maps of the chip wafer defect production image are extracted and substituted into GAP global average pooling to obtain the global feature vector of multi-defect labels, and a chip wafer defect source domain feature pool is constructed.

[0031] Based on real-time chip wafer production images, the encoder end of the chip wafer defect multimodal segmentation network is substituted into the image to extract the global feature vector of defects in the real-time chip wafer production images.

[0032] Based on cosine similarity, the similarity between the global feature vector of defects in the real-time chip wafer production image and each source domain feature vector in the chip wafer defect source domain feature pool is calculated and filtered to obtain the defect source domain map vector matching the real-time chip wafer production image.

[0033] Preferably, step S3 further includes:

[0034] Using Bayesian priors, the mask knowledge of matching defect source domain graph vectors in real-time chip wafer production images is verified and used as prior information of the global feature vector of defects in real-time chip wafer production images in the chip wafer defect multimodal segmentation network. Through the attention mechanism, the similarity between the global feature vector of defects in real-time chip wafer production images and the matching defect source domain graph vectors is calculated and used as the weight of the global feature vector of defects in real-time chip wafer production images.

[0035] The weighted sum of the global feature vector of defects in real-time chip wafer production images and the defect source domain map vector matched in real-time chip wafer production images is used to obtain the auxiliary feature vector for defect segmentation in real-time chip wafer production images.

[0036] By using additive concatenation, the auxiliary feature vector for defect segmentation of real-time chip wafer production images is concatenated with the global feature vector for defects in real-time chip wafer production images. This concatenation is then fed into a multimodal segmentation network for chip wafer defects, outputting a probability map for defect segmentation of real-time chip wafer production images. This map is then input into a category-independent segmentation decoder to obtain a binary mask for defect segmentation of real-time chip wafer production images, generating a multi-defect label segmentation vector for real-time chip wafer production images.

[0037] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0038] This invention proposes an adaptive feature enhancement image segmentation scheme, which includes: constructing a multimodal dataset that decouples defect semantics from background structure, and using controllable generation technology to synthesize high-quality enhanced samples, thereby fundamentally improving the completeness and quality of model training data; and then combining the HRNet high-resolution network with a source domain feature dynamic enhancement mechanism, enabling the model to adaptively utilize historical prior knowledge during the inference stage, thereby improving the accuracy, robustness, and generalization ability of chip wafer defect segmentation to unknown scenarios. Attached Figure Description

[0039] Figure 1 This is a flowchart of an image segmentation method for adaptive feature enhancement of chip wafer defects. Detailed Implementation

[0040] The following description is intended to disclose the invention and enable those skilled in the art to implement it. The preferred embodiments described below are merely examples, and other obvious variations will occur to those skilled in the art.

[0041] Reference Figure 1 As shown, an image segmentation method with adaptive feature enhancement for chip wafer defects includes:

[0042] S1. Obtain a database of historical chip wafer production images, filter historical chip wafer production images according to known defect categories in chip wafer production, construct a multimodal dataset of chip wafer production defects, train a defect dual-branch encoder, and generate chip wafer defect semantic vectors and background structure vectors.

[0043] Step S1 specifically includes:

[0044] Based on a historical chip wafer production image database, preprocessing is performed according to image normalization and size standardization to obtain the associated metadata of historical chip wafer production images;

[0045] Based on the metadata associated with historical chip wafer production images, substitute it into One-Hot encoding to obtain the metadata vector associated with historical chip wafer production images;

[0046] Based on the SAM arbitrary object segmentation model, the known defect categories in chip wafer production are taken as input, a cue encoder is constructed with the text description of the known defect categories, a mask encoder is constructed with the spatial information of the known defect categories in the image, a visual backbone network for known defect categories in chip wafer production is pre-trained, and a label vector for known defect categories in chip wafer production is generated.

[0047] UMAP nonlinear dimensionality reduction involves dimensionality reduction of the known defect category label vector and the associated element vector of historical chip wafer production images. Substituting these vectors into a binary scatter plot yields the scatter distribution of the dimensionality-reduced vector of the known defect category label vector and the scatter distribution of the dimensionality-reduced vector of the associated element vector of historical chip wafer production images.

[0048] Based on DBSCAN density clustering, the K-distance maps of the scatter distribution of dimensionality-reduced vectors of known defect category labels in chip wafer manufacturing and the scatter distribution of dimensionality-reduced vectors of associated elements in historical chip wafer manufacturing images are calculated. The minimum distance and number of neighbors between each defect category cluster label and the metadata cluster label are determined for clustering. This yields the defect category cluster to which the known defect category vectors in chip wafer manufacturing belong, and the metadata cluster label to which the associated element vectors in historical chip wafer manufacturing images belong.

[0049] Based on the chi-square test, the defect category clusters to which the known defect category vectors of chip wafer production belong are used as rows, and the metadata cluster labels to which the attached element vectors of historical chip wafer production images belong are used as tags. The independence probabilities of row clusters and column clusters are calculated to determine the association between the known defect categories of chip wafer production and the attached element vectors of historical chip wafer production images, thus constructing a multimodal dataset of chip wafer production defects.

[0050] Step S1 also includes:

[0051] Based on the ResNet34 convolutional neural network, the global average pooling layer and fully connected layer are removed to construct a backbone network for extracting defect contribution features. The multimodal dataset of chip wafer production defects is used as the input of the convolutional layer to generate low-level feature maps of each modality of chip wafer production defects.

[0052] The backbone is extracted based on the defect contribution features. A defect semantic branch is constructed using an MLP multilayer perceptron, and a background structure branch is constructed using a U-Net network. A defect dual-branch encoder is constructed. The low-level feature maps of each modality of chip wafer production defects are used as input to extract the defect information and the probability of each pixel value position as background in the low-level feature maps of each modality of chip wafer production defects. The intermediate feature maps and background probability maps of each modality of chip wafer production defects are obtained.

[0053] Based on the intermediate feature maps of various modalities of chip wafer manufacturing defects, the defects are substituted into the defect segmentation decoder. The global average pooling layer is used to extract the binary defect masks of various modalities of chip wafer manufacturing defects. The attention weights of the binary defect masks of each modality are obtained by downsampling to the same spatial size as the intermediate feature maps of various modalities of chip wafer manufacturing defects. Weighted global average pooling is then performed on the intermediate feature maps of various modalities of chip wafer manufacturing defects to generate the semantic vector of chip wafer defects.

[0054] Based on the background probability maps of various modalities of chip wafer manufacturing defects, a class-independent segmentation decoder is introduced. Using a global average pooling layer, the modal background probability maps are binarized into background region masks. The high-level semantic feature maps of chip wafer defects are extracted from the defect contribution feature extraction backbone network. The modal background probability maps are then binarized into background region masks and downsampled to the same size as the high-level feature maps. The attention weights of the background region masks are then weighted and global average pooled to generate the background structure vector of chip wafer defects.

[0055] When using it, please refer to the steps outlined above:

[0056] As a further development, a high-quality multimodal dataset was constructed by fusing visual defect features with production metadata and using statistical tests to select strongly correlated samples. A dual-branch encoder was designed, which focuses on defect regions to generate defect semantic vectors through a defect semantic branch, and simultaneously learns image background information to generate background structure vectors through a background structure branch, thus achieving explicit decoupling between defect features and background features. This generated features that are both information-pure and highly correlated, providing a solid foundation for subsequent segmentation tasks and significantly improving the model's adaptability to complex production environments and its robustness in segmentation.

[0057] Furthermore, as an embodiment of step S1:

[0058] Scenario: Data processing and encoder training for oxide layer scratch defects;

[0059] Data preprocessing and metadata acquisition:

[0060] Retrieve 10,000 wafer manufacturing images with a size of 512x512 from the historical database.

[0061] Read the associated metadata for each image, including: process stage: oxidation, equipment chamber ID: FURNACE-03, temperature parameter: 1050℃.

[0062] Metadata vectorization:

[0063] Metadata is processed using One-Hot encoding. For example, metadata [Oxidation, FURNACE-03, 1050℃] is converted into a 15-dimensional sparse vector [0,1,0, ..., 1, ..., 0,1].

[0064] Defect label vector generation:

[0065] The known defect category “oxide scratch” and its textual description “a thin, linear, penetrating oxide layer damage” are input into the pre-trained SAM cue encoder.

[0066] Meanwhile, the marked "oxide scratch" areas in 10,000 images are used as masks and input into the mask encoder of SAM.

[0067] Using the SAM visual backbone network, a 256-dimensional "oxide scratch defect label vector" is generated for each image containing "oxide scratch".

[0068] Cluster analysis and association validation:

[0069] UMAP was used to reduce the 15-dimensional metadata vector and the 256-dimensional defect label vector to 2 dimensions.

[0070] On the dimensionality-reduced scatter plot of metadata, DBSCAN clusters into three clusters, denoted as metadata cluster A (high temperature rapid oxidation), metadata cluster B (standard process), and metadata cluster C (low temperature process).

[0071] On the dimensionality-reduced scatter plot of the defect labels, DBSCAN clusters into two clusters, denoted as defect cluster X (deep and narrow scratches) and defect cluster Y (wide and shallow scratches).

[0072] A chi-square test was performed. The test results showed that the correlation between defect cluster X and metadata cluster A was less than 0.01, which was statistically significant; while the correlation between defect cluster X and metadata clusters B and C was not significant.

[0073] Building a multimodal dataset:

[0074] All image samples in defect cluster X are bound to the metadata vector of their corresponding metadata cluster A to form 500 multimodal data samples of "oxide layer scratch-high temperature rapid oxidation", and then incorporated into the "chip wafer production defect multimodal dataset".

[0075] Training a defective dual-branch encoder:

[0076] The image input defect contribution feature extraction backbone network (based on ResNet34) is used to extract the image input defect contribution features from the above multimodal dataset, and the output is a low-level feature map with a size of 128x128x64.

[0077] Defect semantic branch: Input the low-level feature map into the MLP to generate an intermediate feature map. The defect segmentation decoder outputs a binary mask of "oxide layer scratch", which is downsampled to 128x128 attention weights and weighted global average pooling is applied to the intermediate feature map to finally generate a 128-dimensional "oxide layer scratch semantic vector".

[0078] Background Structure Branch: The low-level feature map is input into U-Net to generate a background probability map. The class-independent segmentation decoder binarizes it into a background region mask. A high-level feature map of size 64x64x256 is extracted from another layer of the backbone network. The background mask is downsampled to 64x64 attention weights, and a weighted global average pooling is performed on the high-level feature map to finally generate a 256-dimensional "oxidation layer background structure vector".

[0079] S2. Based on the semantic vector of chip wafer defects and the background structure vector, a controllable hybrid chip wafer defect generator is constructed to generate a chip wafer defect superposition state sample dataset, train a chip wafer defect multimodal segmentation network, and generate multi-defect label segmentation vectors for chip wafer production images.

[0080] Step S2 specifically includes:

[0081] Based on the CGAN conditional generative adversarial network architecture, a controllable hybrid chip wafer defect generator is established according to SPADE spatial adaptive normalization. The chip wafer defect background structure vector is used as the generated image basis, and the chip wafer defect semantic vector is used as the defect type generation condition. According to the image basis and defect type generation conditions, a defect heatmap and random defect center positions are generated. The distance of each pixel in the defect heatmap to the random defect center position is calculated by Euclidean distance. The distance is substituted into a two-dimensional Gaussian function to obtain the intensity value of each random defect center position in the defect heatmap, thus obtaining the chip wafer production image defect spatial location heatmap.

[0082] The spatial location heatmap of defects in chip wafer manufacturing images and the multimodal dataset of chip wafer manufacturing defects are substituted into the multi-defect task discriminator. The least squares loss is used as the adversarial loss function of the multimodal task discriminator to output the probability that the defects in the chip wafer manufacturing images are the true data distribution. The multi-class cross-entropy loss is used as the auxiliary segmentation function to verify and predict the image segmentation mask in the input multimodal dataset of chip wafer manufacturing defects. This mask serves as the constraint condition for the generated image segmentation structure of the controllable hybrid chip wafer defect generator, generating chip wafer defect superposition state image-mask pairs and constructing a chip wafer defect superposition state sample dataset.

[0083] Step S2 also includes:

[0084] Based on the chip wafer defect superposition state sample dataset and the chip wafer production defect multimodal dataset, an original training set for chip wafer defects was constructed.

[0085] Based on the HRNet high-resolution network, a backbone network for multimodal segmentation of chip wafer defects is established.

[0086] The original training set of chip wafer defects is substituted into the backbone network of the chip wafer defect multimodal segmentation network. The chip wafer defect category and defect background of the last convolutional layer of the network are input into the Sigmoid activation function, and the chip wafer defect multi-class probability tensor is output.

[0087] Based on binary cross-entropy, a pixel-wise multi-task binary cross-entropy loss is established. The loss function between the mask pair of the chip wafer defect multi-class probability tensor output by the backbone network of the chip wafer defect multi-modal segmentation network and the chip wafer production defect image mask in the chip wafer production defect multi-modal dataset is verified. The pixel-wise multi-task binary cross-entropy loss is minimized by gradient descent to obtain the multi-defect label segmentation vector of the chip wafer production image.

[0088] When using it, please refer to the steps outlined above:

[0089] As a further development, by utilizing the S1-decoupled defect semantics and background structure vectors, a SPADE-based conditional generative adversarial network is used to controllably synthesize superimposed samples with highly diverse defect locations and morphologies. Then, this high-quality augmented data is used to train the HRNet high-resolution segmentation network, leveraging its parallel multi-scale fusion capability to preserve details and optimizing it through pixel-by-pixel multi-task loss, ultimately outputting accurate multi-defect label segmentation vectors. This addresses the problem of insufficient rare defect samples and, through the combination of controllable data augmentation and the segmentation network, significantly improves the model's segmentation accuracy and robustness for complex, multi-defect scenes.

[0090] Furthermore, as an embodiment of step S2:

[0091] Controllable generator synthesizes data:

[0092] Take a background structure vector representing "PVD process chamber A" and a defect semantic vector representing "particulate contamination" from step S1, and input them into the controllable hybrid chip wafer defect generator.

[0093] The generator first generates a 128x128 defect heatmap by fusing conditions through the SPADE layer, and then randomly initializes 15 defect center positions in the heatmap.

[0094] Using each center location as the mean, the Euclidean distance from each pixel to the center on the heatmap is calculated, and then substituted into a two-dimensional Gaussian function with a standard deviation of 3 to calculate a heatmap of the spatial location of defects with 15 Gaussian distributions superimposed. The bright areas in the image represent the possible locations where particles may appear.

[0095] Based on the heatmap and conditional vector, the generator synthesizes a 512x512 superposition image with multiple randomly distributed "particulate contamination" defects in the background of "PVD process chamber A" and its corresponding binary mask.

[0096] The multi-defect task discriminator simultaneously evaluates the authenticity of the generated image, performs segmentation verification on the generated mask, calculates adversarial loss and segmentation loss, and inversely optimizes the generator.

[0097] Building the training set and training the segmentation network:

[0098] The 5,000 superimposed image-mask pairs of "particle contamination" and "scratches" generated by the above method are merged with the original 2,000 chip wafer production defect multimodal datasets in S1 to form a total of 7,000 images of the original chip wafer defect training set.

[0099] Initialize an HRNet-W32 as the backbone network for the chip wafer defect multimodal segmentation network.

[0100] The images in the training set are input into HRNet, and the last layer of the network outputs a 512x512x3 multi-class probability tensor (the three channels correspond to "background", "particle pollution" and "scratches" respectively).

[0101] For a training image, calculate the pixel-by-pixel multi-task binary cross-entropy loss: that is, calculate the binary cross-entropy loss of each pixel on the three channels and sum them. For example, the ideal output probability of a pixel labeled "particle contamination" should be [background: 0, particle contamination: 1, scratch: 0].

[0102] Using the Adam optimizer, perform 150 training epochs and minimize the total loss function using gradient descent.

[0103] After training, the network can output a 512x512x3 multi-defect label segmentation vector for a new 512x512 wafer image. The three values ​​of each pixel represent the probability that the point belongs to the background, particle contamination, and scratch, respectively, thus achieving accurate pixel-level multi-defect segmentation.

[0104] S3. Based on the encoder end of the chip wafer defect multimodal segmentation network, extract the global feature vector of multi-defect labels of chip wafer production images, construct the chip wafer defect source domain feature pool, perform association matching and filtering on real-time chip wafer production images, obtain the real-time chip wafer production image matching defect source domain map vector, substitute it into the chip wafer defect multimodal segmentation network, and generate the multi-defect label segmentation vector of real-time chip wafer production images.

[0105] Step S3 specifically includes:

[0106] Based on the encoder end of the chip wafer defect multimodal segmentation network, the spatial and semantic information feature maps of the chip wafer defect production image are extracted and substituted into GAP global average pooling to obtain the global feature vector of multi-defect labels, and a chip wafer defect source domain feature pool is constructed.

[0107] Based on real-time chip wafer production images, the encoder end of the chip wafer defect multimodal segmentation network is substituted into the image to extract the global feature vector of defects in the real-time chip wafer production images.

[0108] Based on cosine similarity, the similarity between the global feature vector of defects in the real-time chip wafer production image and each source domain feature vector in the chip wafer defect source domain feature pool is calculated and filtered to obtain the defect source domain map vector matching the real-time chip wafer production image.

[0109] Step S3 also includes:

[0110] Using Bayesian priors, the mask knowledge of matching defect source domain graph vectors in real-time chip wafer production images is verified and used as prior information of the global feature vector of defects in real-time chip wafer production images in the chip wafer defect multimodal segmentation network. Through the attention mechanism, the similarity between the global feature vector of defects in real-time chip wafer production images and the matching defect source domain graph vectors is calculated and used as the weight of the global feature vector of defects in real-time chip wafer production images.

[0111] The weighted sum of the global feature vector of defects in real-time chip wafer production images and the defect source domain map vector matched in real-time chip wafer production images is used to obtain the auxiliary feature vector for defect segmentation in real-time chip wafer production images.

[0112] By using additive concatenation, the auxiliary feature vector for defect segmentation of real-time chip wafer production images is concatenated with the global feature vector for defects in real-time chip wafer production images. This concatenation is then fed into a multimodal segmentation network for chip wafer defects, outputting a probability map for defect segmentation of real-time chip wafer production images. This map is then input into a category-independent segmentation decoder to obtain a binary mask for defect segmentation of real-time chip wafer production images, generating a multi-defect label segmentation vector for real-time chip wafer production images.

[0113] When using it, please refer to the steps outlined above:

[0114] As a further development, a source domain knowledge base containing global feature vectors of historical defects is constructed through the encoder of a pre-trained segmentation network. When processing real-time images, the most relevant feature vectors are retrieved from the knowledge base using cosine similarity as prior knowledge. These vectors are then weighted and fused using an attention mechanism to generate auxiliary features, which are finally fed into the decoder for segmentation. This enables the model to have online adaptive capabilities, significantly improving its generalization performance and segmentation robustness for superimposed defects by dynamically utilizing historical prior knowledge, achieving knowledge transfer and enhancement without retraining.

[0115] Furthermore, as an embodiment of step S3:

[0116] Construct the source domain feature pool:

[0117] The HRNet segmentation network trained with S2 was used to process a multimodal dataset of chip wafer manufacturing defects containing 10,000 historical wafer images.

[0118] Extract the 512x128x128 feature map of each image at the end of the encoder, and generate 10,000 1024-dimensional global feature vectors for multi-defect labels through global average pooling.

[0119] These 10,000 vectors, along with their corresponding images and segmentation masks, are stored together to form a feature pool for the defect source domain of the chip wafer.

[0120] Real-time image matching and feature enhancement:

[0121] During the deployment phase, the production line camera captures a real-time image of chip wafer production, which features a novel underlying circuit pattern rarely seen in the training data.

[0122] The real-time image is then input into the HRNet encoder, which extracts a 1024-dimensional global feature vector of real-time image defects.

[0123] The cosine similarity between this vector and 10,000 vectors in the source domain feature pool is calculated, and the top 5 vectors with the highest similarity are selected as the source domain image vectors for matching defects. The historical images corresponding to these 5 vectors all contain "chemical residue" defects, but the background patterns are different.

[0124] The attention mechanism calculates the weights of the current real-time image features and these five matching vectors, which are [0.1, 0.5, 0.2, 0.15, 0.05]. This indicates that the model considers the second historical sample to be the most important for the current decision.

[0125] The five matching vectors are weighted and summed to obtain a 1024-dimensional auxiliary feature vector for defect segmentation.

[0126] The auxiliary feature vector is concatenated with the original real-time image feature vector to form a 2048-dimensional enhanced feature vector.

[0127] Generate the final segmentation result:

[0128] The enhanced 2048-dimensional feature vector is input into the decoder part of HRNet.

[0129] The decoder outputs a 512x512 defect segmentation probability map, which is then binarized by a class-independent segmentation decoder to finally generate a multi-defect label segmentation vector for an accurate real-time chip wafer production image, accurately identifying the "chemical residue" defect area on the new background.

[0130] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the claimed invention. The scope of protection claimed by the appended claims and their equivalents is defined.

[0131] In addition, the parts of the technical solutions provided in the embodiments of this application that are consistent with the implementation principles of the corresponding technical solutions in the prior art have not been described in detail, so as to avoid excessive elaboration.

[0132] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the invention. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. An image segmentation method with adaptive feature enhancement for chip wafer defects, characterized in that, include: S1. Obtain a database of historical chip wafer production images, filter historical chip wafer production images according to known defect categories in chip wafer production, construct a multimodal dataset of chip wafer production defects, train a defect dual-branch encoder, and generate chip wafer defect semantic vectors and background structure vectors. S2. Based on the semantic vector of chip wafer defects and the background structure vector, a controllable hybrid chip wafer defect generator is constructed to generate a chip wafer defect superposition state sample dataset, train a chip wafer defect multimodal segmentation network, and generate multi-defect label segmentation vectors for chip wafer production images. S3. Based on the encoder end of the chip wafer defect multimodal segmentation network, extract the global feature vector of multi-defect labels from the chip wafer production image, construct a chip wafer defect source domain feature pool, perform association matching and filtering on the real-time chip wafer production image to obtain the real-time chip wafer production image matching defect source domain map vector, substitute it into the chip wafer defect multimodal segmentation network, and generate the multi-defect label segmentation vector of the real-time chip wafer production image, including: Based on the encoder end of the chip wafer defect multimodal segmentation network, the spatial and semantic information feature maps of the chip wafer defect production image are extracted and substituted into GAP global average pooling to obtain the global feature vector of multi-defect labels, and a chip wafer defect source domain feature pool is constructed. Based on real-time chip wafer production images, the encoder end of the chip wafer defect multimodal segmentation network is substituted into the image to extract the global feature vector of defects in the real-time chip wafer production images. Based on cosine similarity, the similarity between the global feature vector of defects in the real-time chip wafer production image and each source domain feature vector in the chip wafer defect source domain feature pool is calculated and filtered to obtain the real-time chip wafer production image matching defect source domain map vector. Using Bayesian priors, the mask knowledge of matching defect source domain graph vectors in real-time chip wafer production images is verified and used as prior information of the global feature vector of defects in real-time chip wafer production images in the chip wafer defect multimodal segmentation network. Through the attention mechanism, the similarity between the global feature vector of defects in real-time chip wafer production images and the matching defect source domain graph vectors is calculated and used as the weight of the global feature vector of defects in real-time chip wafer production images. The weighted sum of the global feature vector of defects in real-time chip wafer production images and the defect source domain map vector matched in real-time chip wafer production images is used to obtain the auxiliary feature vector for defect segmentation in real-time chip wafer production images. By using additive concatenation, the auxiliary feature vector for defect segmentation of real-time chip wafer production images is concatenated with the global feature vector for defects in real-time chip wafer production images. This concatenation is then fed into a multimodal segmentation network for chip wafer defects, outputting a probability map for defect segmentation of real-time chip wafer production images. This map is then input into a category-independent segmentation decoder to obtain a binary mask for defect segmentation of real-time chip wafer production images, generating a multi-defect label segmentation vector for real-time chip wafer production images.

2. The image segmentation method for adaptive feature enhancement of chip wafer defects according to claim 1, characterized in that, Step S1 specifically includes: Based on a historical chip wafer production image database, preprocessing is performed according to image normalization and size standardization to obtain the associated metadata of historical chip wafer production images; Based on the metadata associated with historical chip wafer production images, one-hot encoding is used to obtain the metadata vector associated with historical chip wafer production images. Based on the SAM arbitrary object segmentation model, the known defect categories in chip wafer production are taken as input, a cue encoder is constructed with the text description of the known defect categories, a mask encoder is constructed with the spatial information of the known defect categories in the image, a visual backbone network for known defect categories in chip wafer production is pre-trained, and a label vector for known defect categories in chip wafer production is generated. UMAP nonlinear dimensionality reduction involves dimensionality reduction of the known defect category label vector and the associated element vector of historical chip wafer production images. Substituting these vectors into a binary scatter plot yields the scatter distribution of the dimensionality-reduced vector of the known defect category label vector and the scatter distribution of the dimensionality-reduced vector of the associated element vector of historical chip wafer production images. Based on DBSCAN density clustering, the K-distance maps of the scatter distribution of dimensionality-reduced vectors of known defect category labels in chip wafer manufacturing and the scatter distribution of dimensionality-reduced vectors of associated elements in historical chip wafer manufacturing images are calculated. The minimum distance and number of neighbors between each defect category cluster label and the metadata cluster label are determined for clustering. This yields the defect category cluster to which the known defect category vectors in chip wafer manufacturing belong, and the metadata cluster label to which the associated element vectors in historical chip wafer manufacturing images belong. Based on the chi-square test, the defect category clusters to which the known defect category vectors of chip wafer production belong are used as rows, and the metadata cluster labels to which the attached element vectors of historical chip wafer production images belong are used as tags. The independence probabilities of row clusters and column clusters are calculated to determine the association between the known defect categories of chip wafer production and the attached element vectors of historical chip wafer production images, thus constructing a multimodal dataset of chip wafer production defects.

3. The image segmentation method for adaptive feature enhancement of chip wafer defects according to claim 2, characterized in that, Step S1 also includes: Based on the ResNet34 convolutional neural network, the global average pooling layer and fully connected layer are removed to construct a backbone network for extracting defect contribution features. The multimodal dataset of chip wafer production defects is used as the input of the convolutional layer to generate low-level feature maps of each modality of chip wafer production defects. The backbone is extracted based on the defect contribution features. A defect semantic branch is constructed using an MLP multilayer perceptron, and a background structure branch is constructed using a U-Net network. A defect dual-branch encoder is constructed. The low-level feature maps of each modality of chip wafer production defects are used as input to extract the defect information and the probability of each pixel value position as background in the low-level feature maps of each modality of chip wafer production defects. The intermediate feature maps and background probability maps of each modality of chip wafer production defects are obtained. Based on the intermediate feature maps of various modalities of chip wafer manufacturing defects, the defects are substituted into the defect segmentation decoder. The global average pooling layer is used to extract the binary defect masks of various modalities of chip wafer manufacturing defects. The attention weights of the binary defect masks of each modality are obtained by downsampling to the same spatial size as the intermediate feature maps of various modalities of chip wafer manufacturing defects. Weighted global average pooling is then performed on the intermediate feature maps of various modalities of chip wafer manufacturing defects to generate the semantic vector of chip wafer defects. Based on the background probability maps of various modalities of chip wafer manufacturing defects, a class-independent segmentation decoder is introduced. Using a global average pooling layer, the modal background probability maps are binarized into background region masks. The high-level semantic feature maps of chip wafer defects are extracted from the defect contribution feature extraction backbone network. The modal background probability maps are then binarized into background region masks and downsampled to the same size as the high-level feature maps. The attention weights of the background region masks are then weighted and global average pooled to generate the background structure vector of chip wafer defects.

4. The image segmentation method for adaptive feature enhancement of chip wafer defects according to claim 3, characterized in that, Step S2 specifically includes: Based on the CGAN conditional generative adversarial network architecture, a controllable hybrid chip wafer defect generator is established according to SPADE spatial adaptive normalization. The chip wafer defect background structure vector is used as the generated image basis, and the chip wafer defect semantic vector is used as the defect type generation condition. According to the image basis and defect type generation conditions, a defect heatmap and random defect center positions are generated. The distance of each pixel in the defect heatmap to the random defect center position is calculated by Euclidean distance. The distance is substituted into a two-dimensional Gaussian function to obtain the intensity value of each random defect center position in the defect heatmap, thus obtaining the chip wafer production image defect spatial location heatmap. The spatial location heatmap of defects in chip wafer manufacturing images and the multimodal dataset of chip wafer manufacturing defects are substituted into the multi-defect task discriminator. The least squares loss is used as the adversarial loss function of the multimodal task discriminator to output the probability that the defects in the chip wafer manufacturing images are the true data distribution. The multi-class cross-entropy loss is used as the auxiliary segmentation function to verify and predict the image segmentation mask in the input multimodal dataset of chip wafer manufacturing defects. This mask serves as the constraint condition for the generated image segmentation structure of the controllable hybrid chip wafer defect generator, generating chip wafer defect superposition state image-mask pairs and constructing a chip wafer defect superposition state sample dataset.

5. The image segmentation method for adaptive feature enhancement of chip wafer defects according to claim 4, characterized in that, Step S2 also includes: Based on the chip wafer defect superposition state sample dataset and the chip wafer production defect multimodal dataset, an original training set for chip wafer defects was constructed. Based on the HRNet high-resolution network, a backbone network for multimodal segmentation of chip wafer defects is established. The original training set of chip wafer defects is substituted into the backbone network of the chip wafer defect multimodal segmentation network. The chip wafer defect category and defect background of the last convolutional layer of the network are input into the Sigmoid activation function, and the chip wafer defect multi-class probability tensor is output. Based on binary cross-entropy, a pixel-wise multi-task binary cross-entropy loss is established. The loss function between the mask pair of the chip wafer defect multi-class probability tensor output by the backbone network of the chip wafer defect multi-modal segmentation network and the chip wafer production defect image mask in the chip wafer production defect multi-modal dataset is verified. The pixel-wise multi-task binary cross-entropy loss is minimized by gradient descent to obtain the multi-defect label segmentation vector of the chip wafer production image.

Images