Wafer defect detection system and method based on multi-mode and image classification model

Abstract

The invention discloses a wafer defect detection system and method based on multiple modes and an image classification model, and relates to the technical field of semiconductor manufacturing and detection. The system comprises an object recognition module, a defect recognition module and a post-processing module, wherein the object recognition module obtains object feature vectors and positions through a large vision segmentation model and an image-text alignment model; the defect identification module obtains defect feature vectors through an image classification small model and an image-text alignment model; and the post-processing module outputs a detection result through similarity calculation and threshold judgment. The method is based on the system and comprises three steps of object recognition, defect feature extraction and post-processing judgment. According to the method, the generalization ability of a multi-modal large model and the detection precision of an image classification small model are combined, image and text bimodal information is introduced, the problem of missing detection in a scene with a complex background, rare defects and tiny defects is reduced, the detection robustness and accuracy are improved, the labor cost is reduced, and the detection requirement of an advanced semiconductor process is met.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of semiconductor manufacturing detection, and particularly relates to a wafer defect detection system and method based on multi-modal and image classification models. BACKGROUND

[0002] In the process of semiconductor wafer manufacturing, various defects are inevitably generated, and wafer defect detection is a key link to guarantee product yield and reliability. With the continuous reduction of process nodes, the types of wafer defects are more complex, the sizes are smaller, and the requirements for detection accuracy are greatly improved.

[0003] Existing detection methods mostly rely on optical microscopic images combined with artificially set detection rules. Such methods have problems such as insufficient generalization ability, high false detection rate, poor adaptability, and the like when identifying complex defects, and are difficult to meet the high-precision detection requirements under advanced processes.

[0004] In recent years, artificial intelligence methods such as deep learning have been introduced into the field of wafer defect detection. Compared with traditional methods, AI models can automatically extract defect features from massive data, avoiding the limitations of artificially setting rules, not only improving detection accuracy and recall rate, but also speeding up detection, reducing dependence on human experience, and having good scalability and adaptability.

[0005] With the rapid development of multi-modal large models, they have the ability to process multiple sources of information such as images and text descriptions at the same time, and can establish associations between different modalities. Related technologies have formed a number of patent achievements. Among them, patent (CN118038454A) proposes a wafer defect root cause analysis scheme based on a proprietary large model. By scanning images with an electron microscope, the Deeplabv3+ network is used to extract local defect features, and the ResNet model is used to extract global features. After classification by SVM, the pre-trained proprietary large model is input to generate defect description text to assist human root cause analysis. Although this scheme realizes the combination of feature extraction and text description, it relies too much on the pre-training effect of the proprietary large model and does not include the overall structured data of the wafer. The correlation analysis of defects and the overall wafer is insufficient, and the root cause analysis still needs to be dominated by humans, with limited automation. Patent (CN118967600A) discloses a training and detection method for a wafer defect analysis model. A multi-modal large model is trained using local defect images, structured data, description text, and labels. The defect type and description text are output after inputting the defect image and structured data. This scheme improves classification accuracy by introducing structured data, but the training process requires the construction of a second type of sample containing description text, which has a high data preparation cost. Moreover, it only relies on a single architecture of a multi-modal large model and does not fully integrate the high-precision advantages of a small image classification model, so the precision needs to be improved in the case of small defect detection.

[0006] However, in complex background, rare defect or small defect scenarios, traditional image classification small models still have the problem of insufficient detection accuracy, and single multi-modal large models are difficult to balance detection efficiency. The existing patent solutions have low automation, high data cost, and difficulty in balancing detection efficiency and accuracy, so there is an urgent need for a detection scheme that combines the advantages of both. SUMMARY

[0007] The present application aims to provide a wafer defect detection system and method based on multi-modal and image classification models to solve the problem of missed detection in complex background, rare defect, and small defect scenarios, and to balance detection accuracy, robustness, and detection efficiency to meet the detection needs of advanced semiconductor processes.

[0008] The technical solution to achieve the purpose of the present application is:

[0009] The wafer defect detection system based on multi-modal and image classification model comprises an object recognition module, a defect recognition module and a post-processing module.

[0010] The object recognition module is configured with a large vision segmentation model and an image-text alignment model, and is used to receive a wafer defect image, obtain a segmentation result of each object through the large vision segmentation model, convert the segmentation result into a feature vector aligned with the image-text through the image-text alignment model, and take the minimum bounding rectangle as the position of each object.

[0011] The defect recognition module is configured with an image classification small model and the image-text alignment model, and is used to receive the wafer defect image, predict a defect type and a corresponding score score1 through the image classification small model, construct a prompt word after substituting the defect type into a preset prompt word template, and convert the prompt word into a defect feature vector aligned with the image-text through the image-text alignment model.

[0012] The post-processing module is used to receive the object feature vector, the position and the defect feature vector, calculate a similarity score of the defect feature vector and the feature vector of each object, take a maximum value as score2 and a corresponding position, determine score1 and score2 through a preset classification threshold, and output a defect type and a position or a defect-free determination result.

[0013] Further, the large vision segmentation model is SAM, i.e., a general segmentation model; the image-text alignment model is CLIP, i.e., a contrastive language-image pre-training model; and the image classification small model is ResNet, i.e., a residual network model.

[0014] Further, the method for calculating the similarity of the feature vectors in the post-processing module is cosine similarity. For the cosine similarity, the higher the score is, the more similar the two are. The score is calculated as follows: the higher the score is, the more similar the two are.

[0015]

[0016] wherein represents a feature vector of a defect, represents a feature vector of each object in an image, represents an inner product of a vector, and represents a norm of a vector, and n is the number of objects segmented when the SAM is used to segment the image.

[0017] Further, the classification threshold preset in the post-processing module is a threshold set set according to defect types, including a first threshold and a second threshold corresponding to each defect type; after receiving the defect type, the score score1 output by the defect recognition module and the maximum similarity score score2 output by the object recognition module, the post-processing module selects the corresponding first threshold and second threshold from the threshold set according to the defect type, compares the score1 with the first threshold and the score2 with the second threshold, and synthesizes the two comparison results according to the preset logical relationship to obtain the final defect determination result.

[0018] Further, the defect recognition module not only constructs a first prompt word for representing the presence of the defect type by substituting the defect type into the preset prompt word template, but also constructs a second prompt word for representing the absence of the defect type in the wafer defect image; the image-text alignment model encodes the first prompt word and the second prompt word respectively to obtain a positive defect feature vector and a negative defect feature vector, and the post-processing module subtracts the similarity between each object feature vector and the negative defect feature vector from the similarity between each object feature vector and the positive defect feature vector to obtain the similarity score of the object, and selects the object with the maximum similarity score from all objects as the score2 and its corresponding position in claim 1.

[0019] Further, the post-processing module is further configured to:

[0020] Calculate at least one geometric feature of each object segmentation result output by the object recognition module, including area, aspect ratio and perimeter; call the geometric feature range corresponding to the defect type according to the defect type, and filter the objects, determine the object with the highest similarity score and the geometric feature within the range as the defect position; when the geometric feature of the object with the highest similarity score is not within the range, reduce the similarity score of the object and reselect the target object or determine it as no defect.

[0021] The wafer defect detection method based on multi-modal and image classification model is suitable for a wafer defect detection system based on multi-modal and image classification model, and includes the following steps:

[0022] Step S1, input the wafer defect image into the object recognition module, obtain the segmentation result of each object through the large visual segmentation model, and then convert the segmentation result into a feature vector after image-text alignment through the image-text alignment model, and record the position of each object;

[0023] Step S2, input the wafer defect image into a defect recognition module, obtain a defect type and a corresponding score score1 through image classification small model prediction, substitute the defect type into a preset prompt word template to construct a prompt word, and convert the prompt word into a defect feature vector aligned with text through the image-text alignment model;

[0024] Step S3, calculate the similarity scores of the defect feature vector and each object feature vector through a post-processing module, take the maximum value as score2 and the corresponding position, compare score1 and score2 with a preset classification threshold value respectively, if any score is greater than the threshold value, output the defect type and the position, and if neither score is greater than the threshold value, determine that there is no defect.

[0025] Compared with the prior art, the technical scheme of the present application has the following beneficial effects:

[0026] (1) The wafer defect detection system and method based on a multi-modal and image classification model proposed in the present application, the defect automatic detection method based on an AI model, breaks away from the traditional optical detection mode relying on manual rule setting, realizes automatic learning of defect features based on data driving, greatly improves the detection accuracy and dependence on manual experience, and greatly reduces the labor cost.

[0027] (2) Introducing image information and text description into wafer defect detection at the same time, not only relying on visual features, but also providing auxiliary information through language modal, thereby reducing the missed detection problem in complex background, rare defect or micro defect scene, and improving the detection ability of the model.

[0028] (3) Utilize the cross-modal understanding ability of the multi-modal large model, combine the lightweight image classification small model, and the two form a complement: the large model provides robustness and generalization ability, and the small model ensures the accuracy of defect detection, so as to balance the detection accuracy and stability. BRIEF DESCRIPTION OF DRAWINGS

[0029] Figure 1 The wafer defect detection overall structure and working process schematic diagram based on the multi-modal and image classification model proposed in the present application;

[0030] Figure 2 The wafer defect detection specific application architecture schematic diagram based on the multi-modal and image classification model proposed in the present application;

[0031] Figure 3 The wafer defect detection system schematic diagram based on the multi-modal and image classification model proposed in the present application. DETAILED DESCRIPTION

[0032] With reference to the accompanying drawings, the technical solutions in the embodiments of the present application will be described clearly and completely. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the protection scope of the present application.

[0033] The wafer defect detection system based on the multi-modal and image classification model comprises an object recognition module, a defect recognition module and a post-processing module, the overall structure and working process are as shown in Figure 1 , and the specific application architecture is as shown in Figure 2 .

[0034] The object recognition module is configured with a large vision segmentation model and an image-text alignment model, and is used to receive a wafer defect image, obtain segmentation results of each object through the large vision segmentation model (i=1~n), and convert the into a feature vector after image-text alignment through the image-text alignment model (i=1~n), and take the minimum bounding rectangle box of the as the position of each object (i=1~n).

[0035] The specific configuration of the large vision segmentation model is as follows:

[0036] When the SAM model is used: the input image is resized to a resolution of 512*512.

[0037] 1024*1024, the default panoramic segmentation mode is enabled, all objects in the image are segmented, and no prompt is used for prompting; when other large vision segmentation models such as SEEM and HQ-SAM are used, the configuration logic is consistent with that of SAM, so as to realize the goal of segmenting all objects in the image.

[0038] The defect recognition module is configured with an image classification small model and the image-text alignment model, and is used to receive the wafer defect image, obtain a defect type and a corresponding score score1 through the image classification small model, construct a large language after the defect type is substituted into a preset large language template, and convert the large language into a defect feature vector after image-text alignment through the image-text alignment model .

[0039] The specific configuration of the image classification small model and the image-text alignment model is as follows:

[0040] When using ResNet50 as a small image classification model: the input image is first resized to a resolution of 256x256, and then cropped to a resolution of 224x224 through the centercrop operation; when using other small image classification models such as SqueezeNet and EfficientNet, the configuration logic is the same as that of ResNet50, with the goal of predicting defect types.

[0041] When CLIP is used as the image-text alignment model: ResNet50 is selected as the image encoder. The input image is first resized to a resolution of 256x256 and then cropped to a resolution of 224x224 through the centercrop operation. The maximum number of tokens for the text encoder is 77. When using other image-text alignment models such as BLIP and FILIP with BPE encoding, the configuration logic is the same as CLIP, with the goal of achieving image-text feature alignment.

[0042] The post-processing module is used to receive the object feature vector. (i=1~n), position (i=1~n) and defect feature vector ,calculate With each The similarity scores scorei (i=1~n) are used, and the maximum value is taken as score2 and the corresponding location is located. The score1 and score2 are judged by a preset classification threshold, and the defect type and location or no defect judgment result is output.

[0043] The initial adjustment value of the preset classification threshold is 0.7 (determined based on real wafer inspection scenario data), which can be adaptively adjusted according to the actual application scenario or data distribution.

[0044] Furthermore, the large visual segmentation model is SAM, i.e., a general segmentation model; the image-text alignment model is CLIP, i.e., a contrastive language-image pre-trained model; and the small image classification model is ResNet, i.e., a residual network model.

[0045] Furthermore, the post-processing module calculates feature vector similarity using cosine similarity. For cosine similarity, the similarity score... The calculation is shown below; a higher score indicates greater similarity between the two:

[0046]

[0047] in Feature vectors representing defects These represent the feature vectors of each object in the image. Represents the dot product of vectors. and norm of the representative vector, n is the number of objects segmented when using SAM to segment the image.

[0048] Further, the classification threshold preset in the post-processing module is a threshold set set according to defect types, including a first threshold and a second threshold corresponding to each defect type; after receiving the defect type, the score score1 output by the defect identification module and the maximum similarity score score2 output by the object identification module, the post-processing module selects the corresponding first threshold and second threshold from the threshold set according to the defect type, compares the score1 with the first threshold, compares the score2 with the second threshold, and synthesizes the two comparison results according to the preset logical relationship to obtain the final defect determination result.

[0049] Further, the defect identification module not only constructs a first prompt word for representing the existence of the defect type by substituting the defect type into the preset prompt word template, but also constructs a second prompt word for representing the non-existence of the defect type in the wafer defect image; the image-text alignment model encodes the first prompt word and the second prompt word respectively to obtain a positive defect feature vector and a negative defect feature vector, and the post-processing module subtracts the similarity between each object feature vector and the negative defect feature vector from the similarity between the object feature vector and the positive defect feature vector as the similarity score of the object, and selects the object with the highest similarity score among all objects as the score2 and its corresponding position in claim 1.

[0050] Further, the post-processing module is further configured to:

[0051] Calculate at least one geometric feature of the area, aspect ratio and perimeter of each object segmentation result output by the object identification module; call the geometric feature range corresponding to the defect type according to the defect type, and filter the objects, and determine the object with the highest similarity score and the geometric feature within the range as the defect position; when the geometric feature of the object with the highest similarity score is not within the range, reduce the similarity score of the object and reselect the target object or determine that there is no defect.

[0052] The wafer defect detection method based on multi-modal and image classification model is suitable for the wafer defect detection system based on multi-modal and image classification model, and the system module is as shown in Figure 3 The steps are as follows:

[0053] Step S1, input the wafer defect image into the object identification module, and obtain the segmentation result of each object through the large visual segmentation model (i=1~n), and then the image-text alignment model is used to align the... Converted to image-text aligned feature vector (i=1~n), and record the position of each object. (i=1~n);

[0054] Step S2: Input the wafer defect image into the defect recognition module, predict the defect type and corresponding score using the image classification mini-model, substitute the defect type into a preset large language template to construct a large language, and convert the large language into an image-text aligned defect feature vector using the image-text alignment model. ;

[0055] Step S3: Calculate using the post-processing module With each The similarity scores scorei (i=1~n) are used, and the maximum value is taken as score2 and the corresponding location location. Score1 and score2 are compared with the preset classification threshold respectively. If either score is greater than the threshold, the defect type and location location are output; if neither is greater than the threshold, it is determined that there is no defect.

[0056] Example 1:

[0057] Inspecting scratch defects on 14nm process wafers:

[0058] Input: Wafer image taken with a scanning electron microscope (SEM);

[0059] Object recognition module: Employs the HQ-SAM model, with configuration logic consistent with SAM: resizes the input image to a resolution of 1024x1024, enables the default panoramic segmentation mode, and obtains segmentation results for four object regions. );

[0060] The CLIP model is used, configured as follows: the image encoder uses ResNet50, the image region corresponding to each mask is first resized to 256x256, then centercropped to 224x224, aligned with the "wafer surface object", and a 1024-dimensional object feature vector is generated. .

[0061] Record the position of the minimum bounding rectangle of each mask. (Coordinate format: x1, y1, x2, y2).

[0062] The defect identification module adopts a ResNet50 model and is configured to: resize an input image to 256x256 and then centercrop to 224x224, predict a defect type as "scratch", and obtain a corresponding score score1=0.88;

[0063] The "scratch" is substituted into a preset prompt word template to obtain a prompt word "This is a wafer image taken by a scanning electron microscope. Please find out the scratch in the image", and the CLIP model (a text encoder with a maximum token number of 77 and BPE encoding) is used to convert the prompt word into a 1024-dimensional defect feature vector embed;

[0064] The post-processing module adopts a cosine similarity to calculate a similarity score between the embed and , and the similarity scores are 0.72, 0.86, 0.68 and 0.75 respectively, and the maximum value score2=0.86 is taken.

[0065] A preset classification threshold value 0.8 is adopted, both score1 and score2 are greater than the threshold value, and a defect type scratch and a position locate2(x1=230, y1=180, x2=350, y2=200) are output.

[0066] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, but not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A wafer defect detection system based on a multimodal and image classification model, characterized in that, It includes an object recognition module, a defect recognition module, and a post-processing module; The object recognition module is equipped with a large visual segmentation model and an image-text alignment model. It is used to receive wafer defect images, obtain the segmentation results of each object through the large visual segmentation model, and then convert the segmented results into image-text aligned feature vectors through the image-text alignment model. The minimum bounding rectangle is used as the position of each object. The defect recognition module is configured with an image classification mini-model and an image-text alignment model. It is used to receive the wafer defect image, predict the defect type and corresponding score1 through the image classification mini-model, construct prompt words by substituting the defect type into a preset prompt word template, and then convert the prompt words into a defect feature vector after image-text alignment through the image-text alignment model. The post-processing module is used to receive object feature vectors, positions, and defect feature vectors, calculate the similarity scores between the defect feature vectors and the feature vectors of each object, take the maximum value as score2 and the corresponding position, determine score1 and score2 by a preset classification threshold, and output the defect type and position or the no-defect determination result.

2. The wafer defect detection system based on a multimodal and image classification model according to claim 1, characterized in that, The large visual segmentation model is SAM, which stands for General Segmentation Model.

3. The wafer defect detection system based on the multimodal and image classification model according to claim 1, characterized in that, The image-text alignment model is CLIP, which stands for Contrastive Language-Image Pre-trained Model.

4. The wafer defect detection system based on a multimodal and image classification model according to claim 1, characterized in that, The image classification mini-model is ResNet, which is a residual network model.

5. The wafer defect detection system based on a multimodal and image classification model according to claim 1, characterized in that, The post-processing module uses cosine similarity to calculate the similarity of feature vectors.

6. The wafer defect detection system based on a multimodal and image classification model according to claim 5, characterized in that, For cosine similarity, the similarity score The calculation is shown below; a higher score indicates greater similarity between the two: ， in Feature vectors representing defects These represent the feature vectors of each object in the image. Represents the dot product of vectors. and The norm of the vector is represented by n, which is the number of objects segmented when using SAM to segment the image.

7. The wafer defect detection system based on a multimodal and image classification model according to claim 1, characterized in that, The preset classification thresholds in the post-processing module are a set of thresholds set according to defect types, including a first threshold and a second threshold corresponding to each defect type. After receiving the defect type, score1 output by the defect recognition module and the maximum similarity score2 output by the object recognition module, the post-processing module selects the corresponding first threshold and second threshold from the threshold set according to the defect type, compares the score1 with the first threshold, compares the score2 with the second threshold, and combines the two comparison results according to a preset logical relationship to obtain the final defect judgment result.

8. The wafer defect detection system based on a multimodal and image classification model according to claim 1, characterized in that, The defect identification module constructs a first prompt word representing the presence of the defect type by substituting the defect type into a preset prompt word template, and also constructs a second prompt word representing the absence of the defect type in the wafer defect image. The image-text alignment model encodes the first prompt word and the second prompt word respectively to obtain a positive defect feature vector and a negative defect feature vector. The post-processing module subtracts the similarity between the object feature vector and the positive defect feature vector from the similarity between the object and the negative defect feature vector to obtain the similarity score of the object, and selects the object with the highest similarity score among all objects as score2 and its corresponding position as described in claim 1.

9. The wafer defect detection system based on a multimodal and image classification model according to claim 1, characterized in that, When determining the defect location based on the similarity score score2, the post-processing module is also configured to: For each object segmentation result output by the object recognition module, calculate at least one of the geometric features among area, aspect ratio, and perimeter; according to the defect type, call the corresponding geometric feature range to filter the objects, and determine the object with the highest similarity score that is within the range as the location of the defect; If the geometric features of the object with the highest similarity score are not within the specified range, the similarity score of that object is reduced and the target object is reselected or determined to be defect-free.

10. A wafer defect detection method based on a multimodal and image classification model, applicable to the wafer defect detection system based on a multimodal and image classification model as described in any one of claims 1 to 9, characterized in that, Includes the following steps: Step S1: Input the wafer defect image into the object recognition module, obtain the segmentation results of each object through the large vision segmentation model, and then convert the segmentation results into image-text aligned feature vectors through the image-text alignment model, and record the position of each object; Step S2: Input the wafer defect image into the defect recognition module, predict the defect type and corresponding score1 through the image classification small model, substitute the defect type into the preset prompt word template to construct prompt words, and convert the prompt words into image-text aligned defect feature vectors through the image-text alignment model. Step S3: Calculate the similarity score between the defect feature vector and the feature vector of each object through the post-processing module, take the maximum value as score2 and the corresponding position, compare score1 and score2 with the preset classification threshold respectively, if either score is greater than the threshold, output the defect type and position; if neither is greater than the threshold, determine that there is no defect.

Images