Integrated circuit process defect diagnosis and analysis method and device and medium
Through the combination of PandaGPT large language model and modulation module, the problems of multimodal feature fusion and complex background defect detection are solved, high-precision detection and question-and-answer analysis of integrated circuit defects are realized, which reduces the forgetting problem of language model and improves dialogue language model performance.
Patent Information
- Application Number
- CN202411488577.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-24
- Publication Date
- 2024-11-26
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
The existing multimodal large model cannot effectively integrate the feature information of visual and text modes in integrated circuit defect diagnosis, it is difficult to detect pixel-level defects in complex backgrounds, and it loses the ability to understand the content of the problem text, resulting in the language model. Forgotten question.
Using PandaGPT large language model, images and text features are extracted through frozen CLIP encoder and prediction module, modular enhancement is performed by combining visual and text adapters, and modular modules are designed to perform multimodal feature alignment and knowledge Q&A, using Focal Loss and Dice Loss , Cross-Entropy Loss constraint detection and question-and-answer process.
It realizes high-precision integrated circuit defect detection and detailed question-and-answer analysis, which reduces the forgetting problem of language model, supports attribution analysis and correct response to defect areas, and improves the language model performance of dialogue.
Smart Images

Figure CN119027411A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of integrated circuit detection, and in particular to an integrated circuit process defect diagnosis and analysis method, device and medium. Background Art
[0002] Integrated circuit (IC) manufacturing is a complex process of manufacturing electronic components on semiconductor surfaces. Considering that these surfaces have microstructures, such as wafers or chips, any inaccuracy in the process may lead to defects. These defects may appear as scratches, particles, holes, etc., which seriously affect the quality and performance of electronic devices. Therefore, it is necessary to perform intelligent diagnosis on wafer images obtained by scanning electron microscope (SEM) to query the defect knowledge on its surface to ensure the reliability of the manufacturing process and improve the yield rate.
[0003] Historically, the query of IC defect knowledge has relied on traditional inspection methods and early deep learning algorithms. Traditional methods, including statistical, matching, and morphological methods, usually require a lot of manual intervention to define rules, select parameters, and interpret results. Although these methods are relatively effective in dealing with simple defects, they have difficulty in dealing with more complex scenarios. Subsequently, early deep learning algorithms that automatically learn complex patterns from data were applied to inspection tasks, marking a huge shift in integrated circuit defect diagnosis from relying on human expertise to leveraging data-driven learning. However, these algorithms rely heavily on large-scale annotated data and still face the challenges of weak generalization ability and poor adaptability to complex or invisible defect types. And they are unable to conduct in-depth question-answering analysis, which limits their ability to understand complex image content.
[0004] In recent years, multimodal big models have attracted extensive attention due to their powerful data processing and image understanding capabilities. They provide a more comprehensive analysis perspective by mining and integrating data features from different modalities, providing a feasible method for intelligently querying defect knowledge on the wafer surface. Although existing multimodal big models are pre-trained on large datasets, their application in IC defect query tasks is still limited due to the differences between training datasets and the lack of specific knowledge in the IC domain. This data difference hinders the model's ability to detect and analyze fine-grained information of wafer defects and provide detailed knowledge queries. Therefore, integrating wafer defect knowledge in the IC domain into pre-trained multimodal big models to bridge the gap between general tasks and actual IC application scenarios is expected to support comprehensive defect knowledge queries. However, existing methods based on fine-tuning of multimodal big models often face several problems: (1) they cannot effectively fuse and align feature information from multiple modalities; (2) they are difficult to accurately detect pixel-level defects in complex backgrounds; and (3) they lose the ability to perform comprehensive question-answering tasks, resulting in the language model forgetting knowledge information, that is, the model loses the ability to judge and understand the content of the question text.
[0005] Therefore, the present invention aims to solve the following three technical problems: 1) How to efficiently and seamlessly fuse and align high-quality features of visual and textual modalities, thereby integrating semantic information and detail information in multimodal features; 2) How to automatically and effectively detect pixel-level defects in complex wafer backgrounds and improve the ability to detect and classify microstructure surface defects; 3) How to conduct detailed question-and-answer analysis and knowledge reasoning on the defect knowledge of the relevant wafer surface without affecting the prior knowledge of the pre-trained model to solve the problem of forgetting the language model. Summary of the invention
[0006] The object of the present invention is to provide an integrated circuit process defect diagnosis and analysis method, device and medium to solve the problems raised in the above background technology.
[0007] To achieve the above object, the present invention provides the following technical solutions:
[0008] A method for diagnosing and analyzing integrated circuit process defects, comprising: Step 1, collecting wafer images and dividing the wafer images into a training set, a test set, and a validation set; Step 2: Input the training set, validation set, and test set into the large language model PandaGPT for iterative model training to obtain the optimal large language model PandaGPT. The training process includes: Step 2.1, given a wafer image, extract specific text labels above each image from the image; Step 2.2, the frozen CLIP image encoder encodes the query image into an initial vector e img , the frozen CLIP text encoder encodes the text tags on the query image and the integrated defect category label set into the initial vector e respectively. txt and e lab ; Step 2.3, modal enhancement stage: Step 2.3.1, e img and e lab The defect type on the wafer surface is predicted by the prediction module, and then the predicted defect type is used to compare with e img and e txt Perform matrix multiplication to generate the eigenvector V img and V txt ; Step 2.3.2, the visual adapter and the text adapter further generate the feature vector V img and V txtare processed into information-rich image tokens T img and the text token T txt ; Step 2.4, defect detection phase: The image token T obtained in step 2.3 img and the text token T txt Input into the detection head that can automatically detect defects to obtain a supervised detection mask Mask; Step 2.5, knowledge question and answer stage: Step 2.5.1, design a modulation module for aligning multimodal feature information. The modulation module includes Q-Former, a learnable corrector, and a cosine calculation unit. Q-Former constitutes the first part of the modulation module, and the learnable corrector and the cosine calculation unit constitute the second part of the modulation module. The Q-Former of the modulation module uses the attention mechanism to convert the image token T img , text token T txt And the mask token T projected by the detection mask Mask mas Aligned into a unified visual token aT vis ; Step 2.5.2, the second part of the modulation module converts the visual token aT vis 、User query token T que And the mask token T projected by the detection mask Mask mas Connect them as prompt instructions to the pre-trained large language model PandaGPT, and then optimize and fine-tune the pre-trained large language model PandaGPT; Step 2.5.3, establish corpus A of defect-related questions and answers and corpus B of defect-unrelated questions and answers, and train corpus A and corpus B alternately according to the prompt instructions input into the large language model in step 2.5.2, so as to prevent the large language model from tending to retrieve newly embedded knowledge when understanding the questions raised by users, so as to achieve the purpose of optimizing and fine-tuning the large language model PandaGPT; Step 3: Perform defect diagnosis analysis on the wafer image to be inspected through the optimal large language model PandaGPT.
[0009] Furthermore, the step 2.3.1 specifically includes: Step 2.3.1.1, reshape the initial vector e using a linear layer and activation function img and e lab The size is expressed as: , in, represents the transposed matrix of the weight matrix, b irepresents the bias vector, σ represents the ReLU activation function; Step 2.3.1.2, calculate each defect category and The cosine value of the angle between them is used to evaluate their similarity, i represents a defect category, and the cross entropy loss function is used to constrain the selection of the corresponding category. The formula of this process is expressed as: , Among them, Consine represents the cosine similarity calculation, P i Represents the probability value of predicting the i-th defect category, P n Represents the predicted defect category, N represents the number of defect categories, max represents the maximum value operation, Softmax is a function that converts a numerical value into a category probability distribution, and exp represents an exponential function; Step 2.3.1.3, defect category P n Respectively with e img and e txt Multiply them together to get the feature vector V containing rich semantic information img and V txt .
[0010] Furthermore, the visual adapter in step 2.3.2 includes a visual expert system, and the text adapter includes a text expert system. Step 2.3.2 specifically includes: Step 2.3.2.1, by e img The visual expert system is initialized with the guidance of txt The text expert system is initialized with the guidance of , so that it can acquire knowledge from the text model; Step 2.3.2.2, during the training process, the initialized visual expert system and text expert system adaptively update parameters and are respectively compared with the feature vector V containing rich semantic information after each update. img and V txt Interact to effectively control the direction and quality of feature flow; Step 2.3.2.3, the final result generated by the visual adapter is T img The final result generated by the text adapter is T txt , where T img and T txt It can be calculated using the following formula: , , Among them, → represents the feature guidance operation, Concat represents the connection operation, z i , (eimg ) i and (e txt ) i Represent random vectors z and e respectively img and e txt The i-th element of e are pre-trained expert features.
[0011] Furthermore, the working process of the detection head in step 2.4 includes: Step 2.4.1, fuse image tokens T by matrix multiplication img and the text token T txt complementary information; Step 2.4.2, the image token T is transformed into img and the text token T txt The fused features are mapped back to high-dimensional features; Step 2.4.3, normalize the mapped high-dimensional features into a mask image Mask through the softmax function, that is, the defect detection result map. The formula of this process is: , in, Represents that four upsampling operations have been performed. Softmax is a function that converts numerical values into category probability distribution. Represents a matrix multiplication operation.
[0012] Furthermore, the working process of the Q-Former in step 2.5.1 is as follows: Step 2.5.1.1, the bidirectional self-attention mechanism allows T img Absorption mas The semantic information and detail information in the image are used to obtain the visual feature f i~m , thereby promoting interaction within the same modality. The formula for this process is: , in, represents the convolution operation, k1 and k2 represent different kernels, and d k represents the dimension of the feature vector, M img and M mas Represent the attention maps based on the image image and the mask image Mask respectively; Step 2.5.1.2: Align visual features f through cross-attention mechanism i~m and the text token T txt The fine-grained information between them allows the sharing of complementary knowledge across modalities and obtains an integrated unified feature f i~m~t The formula is: , Among them, M i~m~t A multimodal attention map representing the output of the cross-attention mechanism; Step 2.5.1.3, the feedforward network obtains the unified feature f i~m~t Mapped to a high-dimensional space, and output high-quality prompt instructions T by activating a nonlinear layer vis .
[0013] Furthermore, the step 2.5.2 specifically includes: In the second part of the modulation module, a scaling factor a is set, and the value of a is dynamically adjusted by learning the association between instructions; a learnable corrector is used in T vis The initialization is performed under the guidance of , and the value of a is simulated by calculating its similarity with the user query. The calculation process of a is: , Among them, • represents the dot product, ▽f c represents the hint feature of the learnable corrector, || || represents the norm of the vector; Assign a to the user query token T vis The coefficient of the updated visual token aT vis , the mask token T projected by the detection mask Mask mas and user query token T que As the input instruction for fine-tuning the large language model, the format is as follows: , in, Represents the instruction format for fine-tuning a large language model.
[0014] Furthermore, in step 2.5.3, corpus-A includes multiple question-answer pairs, each of which is related to defect type, quantity, location, description and analysis, and corpus-B includes multiple question-answer pairs that are not related to defect knowledge; in step 2.5.3, the corpus-A includes multiple question-answer pairs, each of which is related to defect type, quantity, location, description and analysis, and corpus-B includes multiple question-answer pairs that are not related to defect knowledge; in step 2.5.2, the corpus-B includes multiple question-answer pairs that are not related to defect knowledge; The format of training corpus-A and corpus-B alternately.
[0015] Furthermore, in the training process of step 2, three loss functions, namely Focal Loss, Dice Loss and Cross-Entropy Loss, are used to constrain the detection and question-answering processes. Focal Loss and dice Loss are used to constrain the segmentation and positioning capabilities of the large language model, and Cross-Entropy Loss is used to constrain the classification and question-answering capabilities of the large language model. Focal Loss aims to solve the problem of class imbalance. It introduces a modulation factor γ to reduce the relative loss of correctly classified pixels and focus on difficult-to-classify and misclassified pixels. Its calculation process is as follows: , Among them, L focal Represents the Focal Loss loss function, p i represents the probability that the pixel belongs to the correct category, H and W represent the height and width of the image respectively, log represents the logarithmic function, and γ represents the hyperparameter; Dice Loss aims to maximize the overlap between the output and the actual label, encouraging the model to learn to produce results closer to the ideal segmentation. Its calculation process is as follows: , Among them, L dice represents the Dice Loss function, represents the output of the decoder, represents the true label, H and W represent the height and width of the image respectively; Cross-entropy Loss measures the difference between the predicted and actual categories in the prediction module, as well as the difference between the output text of the large language model and the target text in the question-answering task. The calculation process is as follows: , Among them, L ce represents the Cross-Entropy Loss loss function, c represents the number of categories and token instructions in classification and question answering tasks, and y i represents the true label, p i Represents the label of the predicted category, log represents the logarithmic function; The overall loss function L is: , Among them, α, β, δ, and ε are four hyperparameters.
[0016] The present invention also provides an integrated circuit process defect diagnosis and analysis device, comprising one or more processors, for implementing the integrated circuit process defect diagnosis and analysis method as described above.
[0017] The present invention also provides a readable storage medium on which a program is stored. When the program is executed by a processor, the above-mentioned integrated circuit process defect diagnosis and analysis method is implemented.
[0018] Compared with the prior art, the present invention has the following beneficial effects: The present invention discloses a new multimodal large language model for defect knowledge query in the field of integrated circuits, including defects, analysis, question and answer, etc. It adopts three stages to gradually realize the functions of defect detection and high-quality dialogue. The enhanced feature tokens help the model to automatically perform high-precision detection. Dynamically aligned and corrected tokens fine-tune the large language model, which not only supports the attribution analysis of defect areas and correct responses about defect types, locations, quantities, etc., but also alleviates the forgetting problem of the language model in the dialogue. And the effectiveness of the method of the present invention has been verified on an internal wafer dataset and 100 questions. The method of the present invention provides great convenience for the semiconductor industry and also provides further insights for further research on multimodal large models. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] Figure 1 The present invention is a flow chart of an integrated circuit process defect diagnosis and analysis method.
[0020] Figure 2 This is a framework diagram of the training process of the large language model PandaGPT in the integrated circuit process defect diagnosis and analysis method of the present invention.
[0021] Figure 3 It is a structural schematic diagram of an integrated circuit process defect diagnosis and analysis device of the present invention.
[0022] Figure 4 This is a diagram showing the internal wafer data set in the embodiment.
[0023] Figure 5 1 is a visualized heat map result of four defect types compared with the model method not based on the multimodal large model in the embodiment of the present invention.
[0024] Figure 6 The results of visualized Mask images of four defect types are shown in the examples compared with the model method based on a multimodal large model.
[0025] Figure 7 This is an example diagram of the dialogue for a "hole" type defect according to the present invention.
[0026] Figure 8 This is an example diagram of a dialogue for a "scratch" type defect of the present invention. The problems in red font in the diagram are common problems in the field of integrated circuits that are not closely related to the input image. DETAILED DESCRIPTION
[0027] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0028] like Figure 1 and Figure 2 As shown, a method for diagnosing and analyzing integrated circuit process defects comprises the following steps: Step 1: Collect wafer images and divide the wafer images into training set, test set and validation set.
[0029] Step 2: Input the training set, validation set, and test set into the large language model PandaGPT for iterative model training to obtain the optimal large language model PandaGPT. The training process of the large language model PandaGPT consists of a basic stage for modality enhancement and two functional stages for detection and question answering. The training process includes: Step 2.1, first, given a query image, extract the specific text tags above each image from the image by using Pp-ocrv3 optical character recognition (OCR) technology.
[0030] Step 2.2, the frozen CLIP image encoder encodes the query image into an initial vector e img , the frozen CLIP text encoder encodes the text tags on the query image and the integrated defect category label set into the initial vector e respectively. txt and e lab .
[0031] Step 2.3, when analyzing pixel-level defects on the wafer surface, it is challenging to distinguish complex background and foreground features, which hinders the query of defect-related knowledge. To solve this problem, the present invention develops a modal enhancement stage, which consists of a prediction module and two enhancement branches, aiming to highlight relevant defect features and minimize the impact of irrelevant features. The modal enhancement stage includes: Step 2.3.1 is implemented by the prediction module. The pre-trained encoder captures pixel-level detail features in the latent space, but its repeated downsampling operation leads to the loss of semantic features. In this invention, a prediction module is designed to predict the defect category in the image and enhance the semantic features by embedding the prediction result into the initial vector. Specifically, the present invention constructs and calculates a label set containing all defect categories found in the wafer image to support the automatic classification task. Step 2.3.1 includes: e img and e labThe defect type on the wafer surface is predicted by the prediction module, and then the predicted defect type is used to compare with e img and e txt Perform matrix multiplication to generate the eigenvector V img and V txt Step 2.3.1 The specific steps are as follows: Step 2.3.1.1, apply a linear layer and activation function to reshape the initial vector e img and e lab The size of this can be expressed by the formula: , in, represents the transposed matrix of the weight matrix, b i represents the bias vector and σ represents the ReLU activation function.
[0032] Step 2.3.1.2, calculate each defect category and The cosine value of the angle between them is used to evaluate their similarity, i represents a defect category, and the cross entropy loss function is used to constrain the selection of the corresponding category. The formula of this process is expressed as: ,
[0033] Among them, Consine represents the cosine similarity calculation, P i Represents the probability value of predicting the i-th defect category, P n Represents the predicted defect category, N represents the number of defect categories, max represents the maximum value operation, Softmax is a function that converts a numerical value into a category probability distribution, and exp represents an exponential function; Step 2.3.1.3, defect category P n Respectively with e img and e txt Multiply them together to get the feature vector V containing rich semantic information img and V txt .
[0034] Step 2.3.2 is implemented by a multimodal adapter, which includes a visual adapter and a text adapter. Although the semantic features related to the defect attributes are enhanced, a detailed representation of the defect features is still necessary. In the visual and text enhancement branches, the present invention deploys a prompt learning-based adapter to perform adaptive feature optimization using additional prompts from pre-trained experts. Step 2.3.2 includes: the designed visual adapter and text adapter further process these vectors into information-rich image tokens T img and the text token T txtThe visual adapter includes a visual expert system, and the text adapter includes a text expert system. The specific steps of step 2.3.2 are as follows: Step 2.3.2.1, by e img The visual expert system is initialized with the guidance of txt The text expert system is initialized with the guidance of , so that it can acquire knowledge from the text model.
[0035] Step 2.3.2.2, during the training process, the initialized visual expert system and text expert system adaptively update parameters and are respectively compared with the feature vector V containing rich semantic information after each update. img and V txt interact, effectively controlling the direction and quality of feature flow.
[0036] Step 2.3.2.3, the final result generated by the visual adapter is T img The final result generated by the text adapter is T txt , where T img and T txt It can be calculated using the following formula: , , Among them, → represents the feature guidance operation, Concat represents the connection operation, z i , (e img ) i and (e txt ) i Represent random vectors z and e respectively img and e txt The i-th element of e are pre-trained expert features.
[0037] Step 2.4, in the defect detection stage, manually setting the segmentation threshold is not necessarily optimal, but should be continuously adjusted for specific tasks. The detection head designed by the present invention can autonomously learn the specific threshold of each pixel at the feature position to generate a pixel-level mask Mask. Step 2.4 includes: in the defect detection stage, the image token T output by the multimodal adapter is img and the text token T txt Input into the designed detection head that can automatically detect defects to obtain a supervised detection mask Mask. The specific steps of step 2.4 are as follows: Step 2.4.1, fuse image tokens T by matrix multiplication img and the text token T txt complementary information.
[0038] Step 2.4.2, the image token T is transformed into img and the text token T txt The fused features are mapped back to high-dimensional features.
[0039] In step 2.4.3, the mapped high-dimensional features are normalized into a mask image Mask through the softmax function, which is the defect detection result image. The detection head matches the multimodal information and supervises the detail features, so that defects can be accurately detected in a complex wafer background. The formula for this process is: , in, Represents that four upsampling operations have been performed. Softmax is a function that converts numerical values into category probability distribution. Represents a matrix multiplication operation.
[0040] Step 2.5, in the knowledge question answering stage, the commonly used embedded instruction format for fine-tuning large models based on prompt instructions is:
[0041] However, this embedding format of prompt instructions may cause the language model to forget, that is, the model is dominated by the image input and cannot respond to the question appropriately. For example, the latest defect question answering model AnomalyGPT embeds industrial defect knowledge into the pre-trained model based on the above format, but the fine-tuned large language model can only answer questions related to defects, such as: "Is there a defect on the surface of the image?" and "Where is the defect in the image?". However, it cannot answer general questions that are not closely related to the image, such as: "What impact does this defect have on the production line?" and "What are the core process steps in integrated circuit manufacturing?". This phenomenon shows that when the model obtains new information, it loses its understanding of the original knowledge, thereby reducing the model's ability to effectively analyze general knowledge. Research shows that this phenomenon occurs because the model does not fully judge the relevance between the query image and the user's question. The key to solving this problem and ensuring the model's accurate response is: (1) improving the quality of visual instructions; (2) enhancing the ability to evaluate the relevance of visual prompt instructions and user query instructions; (3) optimizing the training strategy of the corpus.
[0042] In the knowledge question answering stage, the present invention modulates T img , T txt And the token T projected by the mask Mask mas , and the question token T raised by the user que Aligned to a unified visual token aT visThe quality of the prompt affects the understanding and response of the fine-tuning model, so it is necessary to align multimodal features to obtain comprehensive information and reduce the fine-tuning burden. The specific steps are as follows: Step 2.5.1, the present invention designs a modulation module for aligning multimodal feature information, the modulation module includes Q-Former, a learnable corrector and a cosine calculation unit, Q-Former constitutes the first part of the modulation module, and the learnable corrector and the cosine calculation unit constitute the second part of the modulation module. The Q-Former of the modulation module uses the attention mechanism to convert the image token T img , text token T txt And the mask token T projected by the detection mask Mask mas Aligned into a unified visual token aT vis The working process of Q-Former is as follows: Step 2.5.1.1, the bidirectional self-attention mechanism allows T img Absorption mas The semantic information and details in f are used to obtain i~m , thereby promoting interaction within the same modality. The formula for this process is: ,
[0043] in, represents the convolution operation, k1 and k2 represent different kernels, and d k represents the dimension of the feature vector, M img and M mas They represent the attention maps based on the image image and the attention maps based on the mask image Mask respectively.
[0044] Step 2.5.1.2: Align visual features f through cross-attention mechanism i~m and the text token T txt The fine-grained information between them allows the sharing of complementary knowledge across modalities. The integrated unified feature f i~m~t The formula is: ,
[0045] Among them, M i~m~t Multimodal attention map representing the output of the criss-cross attention mechanism.
[0046] Step 2.5.1.3, in order to maintain the semantic consistency between the large language model input and the output of the modality enhancement stage, the feedforward network will obtain the unified feature f i~m~t Mapped to a high-dimensional space, and output high-quality prompt instructions T by activating a nonlinear layer vis .
[0047] Step 2.5.2, the second part of the modulation module converts the visual token aT vis 、User query token T que And the mask token T projected by the detection mask Mask mas The two are connected as prompt instructions input into the pre-trained large language model PandaGPT, and then the pre-trained large language model PandaGPT is optimized and fine-tuned. Specifically, since the content of the user query involves knowledge of different tasks, the relationship between the query and the visual instruction must be evaluated before fine-tuning the large language model. The present invention sets a scaling factor a in the second part of the modulation module, and dynamically adjusts its value by learning the association between the instructions (the higher the value of a, the stronger the association). A learnable corrector is used in T vis The initialization is performed under the guidance of , and the value of a is simulated by calculating its similarity with the user query. The calculation process of a is: ,
[0048] Where, • represents the dot product, ▽f c represents the hint feature of the learnable corrector, and || || represents the norm of the vector.
[0049] Assign a to the user query token T vis The coefficient of the updated visual token aT vis , the mask token T projected by the detection mask Mask mas and user query token T que As the input instruction for fine-tuning the large language model, the format is as follows: ,
[0050] in, Represents the instruction format for fine-tuning a large language model.
[0051] In step 2.5.3, a corpus A of defect-related questions and answers and a corpus B of defect-unrelated questions and answers are established. According to the prompt instructions input into the large language model in step 2.5.2, corpus A and corpus B are trained alternately in a ratio of 2:1 to prevent the large language model from tending to retrieve newly embedded knowledge when understanding the questions raised by users, thereby achieving the purpose of optimizing and fine-tuning the large language model PandaGPT.
[0052] Corpus-A includes multiple question-answer pairs, each of which is related to defect type, quantity, location, description, and analysis. For example, Q: What type of defect is in the image? A: The defect in the image is a hole. Corpus-B includes multiple question-answer pairs that are not related to defect knowledge. For example, Q: What is the capital of China? A: The capital of China is Beijing. Step 2.5.3 is input into the large language model according to step 2.5.2. The format of training corpus-A and corpus-B alternately.
[0053] In the corpus training dedicated to large language model training, the alternating training strategy balances the learning of new and old knowledge.
[0054] During the model training process, three loss functions, Focal Loss, Dice Loss, and Cross-Entropy Loss, are used to constrain the detection and question-answering processes. Focal Loss and dice Loss are used to constrain the segmentation and positioning capabilities of the large language model, and Cross-Entropy Loss is used to constrain the classification and question-answering capabilities of the large language model.
[0055] Focal Loss: Focal Loss aims to solve the problem of class imbalance. It introduces a modulation factor γ to reduce the relative loss of correctly classified pixels and focus on difficult-to-classify and misclassified pixels. Its calculation process is as follows: ,
[0056] Among them, L focal Represents the Focal Loss loss function, p i represents the probability that the pixel belongs to the correct category, H and W represent the height and width of the image respectively, log represents the logarithmic function, and the present invention sets γ to 2.
[0057] Dice Loss: Dice Loss aims to maximize the overlap between the output and the actual label, encouraging the model to learn to produce results closer to the ideal segmentation. Its calculation process is as follows: ,
[0058] Among them, L dice represents the Dice Loss function, represents the output of the decoder, represents the true label, H and W represent the height and width of the image respectively.
[0059] Cross-Entropy Loss: Cross-entropy Loss measures the difference between the predicted and actual categories in the prediction module, as well as the difference between the output text of the language model and the target text in the question-answering task. Its calculation process is as follows: ,
[0060] Among them, L ce represents the Cross-Entropy Loss loss function, c represents the number of categories and token instructions in classification and question answering tasks, and y i represents the true label, p i Represents the label of the predicted category, and log represents the logarithmic function. The overall loss function L is: ,
[0061] By default, the present invention sets all four coefficients to 1.
[0062] Step 3: Perform defect diagnosis analysis on the wafer image to be inspected through the optimal large language model PandaGPT.
[0063] See also Figure 3 The present invention provides an integrated circuit process defect diagnosis and analysis device, which includes one or more processors for implementing an integrated circuit process defect diagnosis and analysis method in the above embodiment.
[0064] An embodiment of an integrated circuit process defect diagnosis and analysis device of the present invention can be applied to any device with data processing capabilities, and the device with data processing capabilities can be a device or apparatus such as a computer. The device embodiment can be implemented through software, or through hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of any device with data processing capabilities in which it is located reading the corresponding computer program instructions in the non-volatile memory into the internal memory for execution. From a hardware perspective, if Figure 3 As shown in FIG. 1 , a hardware structure diagram of an integrated circuit process defect diagnosis and analysis device of the present invention is a device having data processing capability, except Figure 3 In addition to the processor, memory, network interface, and non-volatile memory shown, any device with data processing capabilities in the embodiments may also include other hardware, usually based on the actual functions of the device with data processing capabilities, which will not be described in detail.
[0065] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.
[0066] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0067] The present invention also provides a readable storage medium on which a program is stored. When the program is executed by a processor, an integrated circuit process defect diagnosis and analysis method in the above embodiment is implemented.
[0068] The readable storage medium may be an internal storage unit of any device with data processing capability described in any of the aforementioned embodiments, such as a hard disk or a memory. The readable storage medium may also be an external storage device, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the readable storage medium may also include both an internal storage unit of any device with data processing capability and an external storage device. The readable storage medium is used to store the computer program and other programs and data required by any device with data processing capability, and may also be used to temporarily store data that has been output or is to be output. Example
[0069] 1. Experimental Dataset: The experiments of this invention are conducted on internal wafer datasets, such as Figure 4 As shown. The dataset includes images with a resolution of 480×480, each of which is accompanied by a corresponding mask image and related text description and analysis. It includes 1226 defect-free wafer images and 1182 wafer images with four common defect types (holes: 250, particles: 500, pattern deformation: 250, scratches: 182). The wafer images in this dataset contain a variety of complex backgrounds, and each defect type exhibits unique morphology and characteristics. The data is divided into training and test sets in a ratio of 7:3, where both groups consist of good and defective images.
[0070] 2. Experimental details: This paper uses PandaGPT as the basic multimodal large model, Vicuna-7B as the language model, and ImageBind Huge as the frozen encoder. During the training process, the AdamW optimizer (β1=0.9, β2=0.999) is used, and the initial learning rate is set to 1e -4 , using the cosine annealing strategy to gradually reduce to 1e -6 The method of the present invention is trained on three 4090Ti GPUs, with the batch size set to 24 and the number of training rounds set to 50.
[0071] 3. Experimental evaluation indicators: In the detection task, four indicators are used to evaluate model performance: image AUC, pixel AUC, per-region overlap (PRO), and average precision (AP). Image AUC and pixel AUC evaluate the model's ability to determine whether there are defects in the image, while PRO and AP measure the model's ability to identify and locate defects. In the question-answering task, we ask 15 questions for each of the four defect types, including inquiries about the existence, category, location, quantity, appearance description, and root cause analysis of the defects. In addition, we asked 50 questions unrelated to defects (general questions related to or unrelated to the integrated circuit field) to verify the forgetting problem of the language model. We use the percentage of correct answers as a relevant indicator to evaluate the model's question-answering ability.
[0072] 4. Experimental results: (1) Quantitative results: The quantitative results report the accuracy of supervised defect detection and the correctness of knowledge question-answering responses. Tables 1, 2, 3, and 4 report the image AUC, pixel AUC, PRO, and AP values of four different methods in an internal wafer dataset. It can be observed that for most defect categories, the method of the present invention outperforms all other methods in the four evaluation metrics. For example, compared with AnomalyGPT, which is also fine-tuned using prompt learning, the method of the present invention achieves an image AUC of 1.85% and a pixel AUC of 3.03%. Compared with the traditional detection model PRN, the method of the present invention improves PRO and AP by 10.12% and 14.91%, respectively.
[0073] Table 5 reports the accuracy of different language models in answering defect-related and unrelated questions, where the invented method achieves state-of-the-art results. For example, compared to the powerful GPT-4, the invented method achieves comparable performance on general questions and improves the accuracy of defect analysis answers by 7.50%. Compared to Lisa, which embeds new tasks into the pre-trained model, the invented method improves the accuracy of answering questions unrelated to the new task by 78.00%.
[0074] Table 1: Results of the image AUC metric for fully supervised experiments on the internal wafer dataset
[0075] Table 2: Results of pixel AUC metric for fully supervised experiments on the internal wafer dataset
[0076] Table 3: Results of PRO metrics for fully supervised experiments on the internal wafer dataset
[0077] Table 4: Results of AP metrics for fully supervised experiments on the internal wafer dataset
[0078] Table 5: Accuracy of different model methods for different questions (%), “-” means the question is not supported
[0079] (2) Qualitative results: Generate a comparison chart of the test results and a dialogue chart of the question-and-answer results on the internal wafer dataset. Use heat maps and mask images to verify the intuitive test results to demonstrate the effectiveness of the proposed method in supervised test tasks. Figure 5 From the heat map in Figure 2, we can see that the proposed method pays more attention to the defective area than other models, while avoiding paying too much attention to the normal area. Figure 6 In the Mask image, compared with Lisa and AnomalyGPT, the proposed method provides more accurate edge segmentation details.
[0080] according to Figure 7 and Figure 8 , the method of the present invention provides a detailed description of defect knowledge, providing engineers with more useful information. Specifically, the method of the present invention is good at identifying defects on the surface of wafer images, locating their locations and distinguishing their specific properties, such as Figure 7 The obvious "holes" and Figure 8 In addition, the method of the present invention can analyze the root cause of wafer defects and propose appropriate solutions from a process perspective, thus becoming an effective compass for improving production lines. And the method of the present invention can also provide accurate answers to general integrated circuit field questions that are not closely related to the input image, such as Figure 8 As shown, for example, the process steps and lithography steps in integrated circuit manufacturing and how artificial intelligence promotes the development of integrated circuit manufacturing are explained. This verifies the effectiveness of the modulation module proposed in the present invention in solving the forgetting of the language model.
[0081] (3) Ablation Experiments: Ablation studies are performed to demonstrate the importance of each component introduced and its operation at each stage.
[0082] (3.1) Stage components: Table 6 confirms the importance of each individual component and the best way to construct the sub-stages. It can be seen that the prediction module and pre-trained experts significantly improve the defect detection ability in the modality enhancement stage, and the operation of the modulation module plays a crucial role in solving the forgetting problem of the language model in the knowledge question answering stage.
[0083] Table 6: Ablation results of important parts at each stage using pixel-AUC and defect-related question answering accuracy (%)
[0084] (3.2) Design of operations in the prediction module: Table 7 shows the advantages of the operation strategy designed for the prediction module by the present invention. It can be seen that using angle cosine to calculate the similarity between two vectors can better evaluate the relationship between them compared with matrix multiplication or other operations. And when used in combination with Linear and ReLU, better calculation effects can be obtained.
[0085] Table 7: Comparison of the effects of different strategies in the prediction module and use the AUC indicator to evaluate the detection accuracy
[0086] (3.3) Prompt instruction embedding scheme: As can be seen from Table 8, the embedding scheme for prompt instructions of the present invention shows the best effect in alleviating the forgetting problem of the language model. By aligning and updating the corresponding coefficients "a" of the visual mark tokens, the method of the present invention can effectively identify the relationship between the user query and the visual input, thereby alleviating the forgetting problem of the language model.
[0087] Table 8: Ablation experiment on the visual cue instruction embedding scheme to verify the knowledge question answering function
[0088] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.
Claims
1. A method for diagnosing and analyzing integrated circuit process defects, It is characterized in that include: Step 1, collecting wafer images and dividing the wafer images into a training set, a test set, and a validation set; Step 2: Input the training set, validation set, and test set into the large language model PandaGPT for iterative model training to obtain the optimal large language model PandaGPT. The training process includes: Step 2.1, given a wafer image, extract specific text labels above each image from the image; Step 2.2, the frozen CLIP image encoder encodes the query image into an initial vector e img , the frozen CLIP text encoder encodes the text tags on the query image and the integrated defect category label set into the initial vector e respectively. txt and e lab ; Step 2.3, modal enhancement stage: Step 2.3.1, e img and e lab The defect type on the wafer surface is predicted by the prediction module, and then the predicted defect type is used to compare with e img and e txt Perform matrix multiplication to generate the eigenvector V img and V txt ; Step 2.3.2, the visual adapter and the text adapter further generate the feature vector V img and V txt are processed into information-rich image tokens T img and the text token T txt ; Step 2.4, defect detection phase: The image token T obtained in step 2.3 img and the text token T txt Input into the detection head that can automatically detect defects to obtain a supervised detection mask Mask; Step 2.5, knowledge question and answer stage: Step 2.5.1, design a modulation module for aligning multimodal feature information. The modulation module includes Q-Former, a learnable corrector, and a cosine calculation unit. Q-Former constitutes the first part of the modulation module, and the learnable corrector and the cosine calculation unit constitute the second part of the modulation module. The Q-Former of the modulation module uses the attention mechanism to convert the image token T img , text token T txt And the mask token T projected by the detection mask Mask mas Aligned into a unified visual token aT vis ; Step 2.5.2, the second part of the modulation module converts the visual token aT vis 、User query token T que And the mask token T projected by the detection mask Mask mas Connect them as prompt instructions to the pre-trained large language model PandaGPT, and then optimize and fine-tune the pre-trained large language model PandaGPT; Step 2.5.3, establish corpus A of defect-related questions and answers and corpus B of defect-unrelated questions and answers, and train corpus A and corpus B alternately according to the prompt instructions input into the large language model in step 2.5.2, so as to prevent the large language model from tending to retrieve newly embedded knowledge when understanding the questions raised by users, so as to achieve the purpose of optimizing and fine-tuning the large language model PandaGPT; Step 3: Perform defect diagnosis analysis on the wafer image to be inspected through the optimal large language model PandaGPT.
2. The integrated circuit process defect diagnosis and analysis method according to claim 1, It is characterized in that The step 2.3.1 specifically includes: Step 2.3.1.1, reshape the initial vector e using a linear layer and activation function img and e lab The size is expressed as: , in, represents the transposed matrix of the weight matrix, b i represents the bias vector, σ represents the ReLU activation function; Step 2.3.1.2, calculate each defect category and The cosine value of the angle between them is used to evaluate their similarity, i represents a defect category, and the cross entropy loss function is used to constrain the selection of the corresponding category. The formula of this process is expressed as: , Among them, Consine represents the cosine similarity calculation, P i Represents the probability value of predicting the i-th defect category, P n Represents the predicted defect category, N represents the number of defect categories, max represents the maximum value operation, Softmax is a function that converts a numerical value into a category probability distribution, and exp represents an exponential function; Step 2.3.1.3, defect category P n Respectively with e img and e txt Multiply them together to get the feature vector V containing rich semantic information img and V txt .
3. The integrated circuit process defect diagnosis and analysis method according to claim 1, It is characterized in that The visual adapter in step 2.3.2 includes a visual expert system, and the text adapter includes a text expert system. Step 2.3.2 specifically includes: Step 2.3.2.1, by e img The visual expert system is initialized with the guidance of txt The text expert system is initialized with the guidance of , so that it can acquire knowledge from the text model; Step 2.3.2.2, during the training process, the initialized visual expert system and text expert system adaptively update parameters and are respectively compared with the feature vector V containing rich semantic information after each update. img and V txt Interact to effectively control the direction and quality of feature flow; Step 2.3.2.3, the final result generated by the visual adapter is T img The final result generated by the text adapter is T txt , where T img and T txt It can be calculated using the following formula: , , Among them, → represents the feature guidance operation, Concat represents the connection operation, z i , (e img ) i and (e txt ) i Represent random vectors z and e respectively img and e txt The i-th element of e are pre-trained expert features.
4. The integrated circuit process defect diagnosis and analysis method according to claim 1, It is characterized in that The working process of the detection head in step 2.4 includes: Step 2.4.1, fuse image tokens T by matrix multiplication img and the text token T txt complementary information; Step 2.4.2, the image token T is transformed into img and the text token T txt The fused features are mapped back to high-dimensional features; Step 2.4.3, normalize the mapped high-dimensional features into a mask image Mask through the softmax function, that is, the defect detection result map. The formula of this process is: , in, Represents that four upsampling operations have been performed. Softmax is a function that converts numerical values into category probability distribution. Represents a matrix multiplication operation.
5. The integrated circuit process defect diagnosis and analysis method according to claim 1, It is characterized in that The working process of Q-Former in step 2.5.1 is as follows: Step 2.5.1.1, the bidirectional self-attention mechanism allows T img Absorption mas The semantic information and detail information in the image are used to obtain the visual feature f i~m , thereby promoting interaction within the same modality. The formula for this process is: , in, represents the convolution operation, k 1 , k 2 Represents different kernels, d k represents the dimension of the feature vector, M img and M mas Represent the attention maps based on the image image and the mask image Mask respectively; Step 2.5.1.2: Align visual features f through cross-attention mechanism i~m and the text token T txt The fine-grained information between them allows the sharing of complementary knowledge across modalities and obtains an integrated unified feature f i~m~t The formula is: , Among them, M i~m~t A multimodal attention map representing the output of the cross-attention mechanism; Step 2.5.1.3, the feedforward network obtains the unified feature f i~m~t Mapped to a high-dimensional space, and output high-quality prompt instructions T by activating a nonlinear layer vis .
6. The integrated circuit process defect diagnosis and analysis method according to claim 1, It is characterized in that The step 2.5.2 specifically includes: In the second part of the modulation module, a scaling factor a is set, and the value of a is dynamically adjusted by learning the association between instructions; a learnable corrector is used in T vis The initialization is performed under the guidance of , and the value of a is simulated by calculating its similarity with the user query. The calculation process of a is: , Among them, • represents the dot product, ▽f c represents the hint feature of the learnable corrector, || || represents the norm of the vector; Assign a to the user query token T vis The coefficient of the updated visual token aT vis , the mask token T projected by the detection mask Mask mas and user query token T que As the input instruction for fine-tuning the large language model, the format is as follows: , in, Represents the instruction format for fine-tuning a large language model.
7. The integrated circuit process defect diagnosis and analysis method according to claim 6, It is characterized in that In step 2.5.3, corpus-A includes multiple question-answer pairs, each of which is related to defect type, quantity, location, description and analysis, and corpus-B includes multiple pairs of questions and answers that are not related to defect knowledge; in step 2.5.3, the corpus-A includes multiple question-answer pairs, each of which is related to defect type, quantity, location, description and analysis, and the corpus-B includes multiple pairs of questions and answers that are not related to defect knowledge; in step 2.5.2, the ... The format of training corpus-A and corpus-B alternately.
8. The integrated circuit process defect diagnosis and analysis method according to claim 1, It is characterized in that In the training process of step 2, three loss functions, namely Focal Loss, Dice Loss and Cross-Entropy Loss, are used to constrain the detection and question-answering processes. Focal Loss and dice Loss are used to constrain the segmentation and positioning capabilities of the large language model, and Cross-Entropy Loss is used to constrain the classification and question-answering capabilities of the large language model. Focal Loss aims to solve the problem of class imbalance. It introduces a modulation factor γ to reduce the relative loss of correctly classified pixels and focus on difficult-to-classify and misclassified pixels. Its calculation process is as follows: , Among them, L focal Represents the Focal Loss loss function, p i represents the probability that the pixel belongs to the correct category, H and W represent the height and width of the image respectively, log represents the logarithmic function, and γ represents the hyperparameter; Dice Loss aims to maximize the overlap between the output and the actual label, encouraging the model to learn to produce results closer to the ideal segmentation. Its calculation process is as follows: , Among them, L dice represents the Dice Loss function, represents the output of the decoder, represents the true label, H and W represent the height and width of the image respectively; Cross-entropy Loss measures the difference between the predicted and actual categories in the prediction module, as well as the difference between the output text of the large language model and the target text in the question-answering task. The calculation process is as follows: , Among them, L ce represents the Cross-Entropy Loss loss function, c represents the number of categories and token instructions in classification and question answering tasks, and y i represents the true label, p i Represents the label of the predicted category, log represents the logarithmic function; The overall loss function L is: , Among them, α, β, δ, and ε are four hyperparameters.
9. An integrated circuit process defect diagnosis and analysis device, It is characterized in that It comprises one or more processors, and is used to implement an integrated circuit process defect diagnosis and analysis method as described in any one of claims 1-8.
10. A readable storage medium, It is characterized in that A program is stored thereon, and when the program is executed by a processor, an integrated circuit process defect diagnosis and analysis method according to any one of claims 1 to 8 is implemented.
Citation Information
Patent Citations
Wafer surface defect regional detection method based on visual image
CN110286126A
SMT defect diagnosis and improvement method and system based on image analysis
CN118552546A
Method and system for semiconductor wafer defect review
US20210232745A1
Computer implemented method for the detection of defects in an imaging dataset of an object comprising integrated circuit patterns, computer-readable medium, computer program product and a system making use of such methods
WO2024160623A1
Cited By
Generative artificial intelligence generated image detection method, system and device and medium
CN119444750A
Apparent defect detection method and system based on multi-modal large model
CN119762485A
Auditing question and answer large model optimization method based on prompt fine tuning
CN120031019A
Knowledge reasoning method and device for integrated circuit wafer process and medium
CN120744140A
A knowledge reasoning method, device and medium for an integrated circuit wafer process
CN120744140B
