A method and system for fast positioning of joint semantic interpretation of semiconductor defects

Abstract

The application discloses a kind of fast positioning combined semantic interpretation's semiconductor defect analysis method and system, belong to semiconductor defect analysis field.Wafer defect image diagnosis data set is obtained, including wafer defect SEM image and diagnosis report, diagnosis report includes sentence for multiple defect diagnosis tasks;Defect image diagnosis network is trained using wafer defect image diagnosis data set, and defect image diagnosis network includes FasterR-CNN encoder and the LSTM decoder based on attention mechanism for different defect diagnosis tasks;In the training process of defect image diagnosis network, the sentence of defect diagnosis task is regarded as the input of decoder, and the target sentence is generated by guiding decoder;Wafer defect SEM image is analyzed using the trained defect image diagnosis network, and generates diagnosis report.The application establishes a closer link between the latest progress of visual and language tasks and target detection, and the accuracy of output sentence is greatly improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of semiconductor defect analysis, and particularly relates to a semiconductor defect analysis method and system for rapid positioning and joint semantic interpretation. BACKGROUND

[0002] In recent years, problems combining image and language understanding, such as image captioning and visual question answering, have attracted a large amount of research in the fields of computer vision and natural language processing. However, in the field of wafer image automatic analysis, the acquisition of semantic information is still a complex and difficult problem. Due to the harshness of industrial production detection, wafer defect image captioning needs to use coherent, concise and professional language to describe and accurately summarize the specific process layer of a specific defect. This not only requires computer vision knowledge to identify the dominant defect and detect the relationship between different defect parts and the background in the same image, but also requires the model to parse the image features identified by the sentence as clearly as possible and as rich in vocabulary as possible. This defect image description task provides more specific and semantic-level information than traditional visual tasks, and provides stronger technical support in cross-modal retrieval, fault warning, diagnosis report generation and other applications.

[0003] The existing research on wafer defect images at the present stage still focuses on defect classification, target recognition, segmentation and other aspects. These works usually only identify defects in defect scanning electron microscope images, obtain defect class labels, and ignore other easily overlooked features of defects, information interaction with the background and the like. The purpose of the present application is to generate a specified comprehensive sentence that summarizes the image content at the semantic level. Compared with traditional encoders, regional features can provide object-level information, identify the most prominent regions in the image and represent them by feature vectors, which can greatly reduce the difficulty of visual semantic embedding. SUMMARY

[0004] To solve the problems in the prior art, the present application provides a semiconductor defect analysis method and system for rapid positioning and joint semantic interpretation.

[0005] The technical solutions adopted by the present application are as follows:

[0006] In a first aspect, the present application discloses a semiconductor defect analysis method for rapid positioning and joint semantic interpretation, comprising:

[0007] obtaining a wafer defect image diagnosis data set containing a wafer defect SEM image and a diagnosis report, wherein the diagnosis report contains a plurality of sentences for different defect diagnosis tasks;

[0008] The defect image diagnosis network is constructed, and the defect image diagnosis network comprises a Faster R-CNN encoder and an attention mechanism-based LSTM decoder for different defect diagnosis tasks; the defect image diagnosis network is trained by using a wafer defect image diagnosis dataset, a wafer defect SEM image is input into the Faster R-CNN encoder, a feature vector of the wafer defect SEM image is obtained, a word vector representation of a corresponding word of a corresponding defect diagnosis task is generated by using the LSTM decoder to decode the feature vector step by step, a word is generated according to the word vector representation, and all the words are sequentially spliced to form a sentence of the defect diagnosis task; and a diagnosis report is obtained by combining sentences of different defect diagnosis tasks.

[0009] In the training process of the defect image diagnosis network, the sentence of the defect diagnosis task is used as the input of the LSTM decoder to guide the LSTM decoder to generate a target sentence.

[0010] The trained defect image diagnosis network is used to analyze a wafer defect SEM image to be processed, and a diagnosis report is generated.

[0011] Further, the defect diagnosis task comprises:

[0012] a defect position, shape and size diagnosis task;

[0013] a defect concave-convex and deformation degree diagnosis task;

[0014] a defect process layer diagnosis task;

[0015] a defect background and composition information diagnosis task;

[0016] a defect category diagnosis task;

[0017] The defect diagnosis task at least comprises one of the above tasks.

[0018] Further, the sentence of the defect diagnosis task in the diagnosis report simultaneously satisfies the following conditions:

[0019] Except for the sentence corresponding to the defect category diagnosis task, at least six words should be contained in one sentence;

[0020] Ambiguous words and grammar are not used;

[0021] The sentence content contains all defects.

[0022] Further, the wafer defect SEM image in the wafer defect image diagnosis dataset is obtained by synthesizing wafer defect SEM images taken at different angles.

[0023] Further, before training the defect image diagnosis network, the sentences in the diagnosis report need to be preprocessed, including:

[0024] Converting all capital letters in the sentence to lowercase letters, removing all punctuation marks in the sentence and performing word segmentation, converting each sentence into a list containing multiple words, and saving the processing result in a dictionary, the key of the dictionary being the ID of the wafer defect SEM image, and the value of the dictionary being a list containing multiple words;

[0025] All sentences in the diagnosis report are preprocessed in the same way.

[0026] In a second aspect, the present application discloses a semiconductor defect analysis system for rapid positioning and joint semantic interpretation, comprising:

[0027] A data acquisition module is configured to acquire a wafer defect image diagnosis data set in a training phase, the wafer defect image diagnosis data set including a wafer defect SEM image and a diagnosis report, the diagnosis report including multiple sentences for different defect diagnosis tasks; and acquire a wafer defect SEM image to be analyzed in a prediction phase;

[0028] A defect image diagnosis network module includes a Faster R-CNN encoder and an LSTM decoder based on an attention mechanism for different defect diagnosis tasks; the wafer defect SEM image is input into the Faster R-CNN encoder to obtain a feature vector of the wafer defect SEM image, the LSTM decoder is used to decode the feature vector step by step to generate a word vector representation of the words corresponding to the corresponding defect diagnosis task, the words are generated according to the word vector representation, and all the words are sequentially spliced to form a sentence of the defect diagnosis task; the sentences of different defect diagnosis tasks are combined to obtain a diagnosis report;

[0029] A network training module is configured to train the defect image diagnosis network using the wafer defect image diagnosis data set, and in the training process of the defect image diagnosis network, the sentences of the defect diagnosis task are input into the LSTM decoder to guide the LSTM decoder to generate target sentences;

[0030] A defect analysis module is configured to analyze the wafer defect SEM image to be processed using the trained defect image diagnosis network to generate a diagnosis report.

[0031] Compared with the prior art, the present application has the following beneficial effects:

[0032] (1) Faster-Rcnn is applied to the encoder in the field of image caption processing, which enables attention to be calculated on objects and other important image regions. In the method of the present application, the image region of the selected target frame has an associated feature vector for each region, and the feature weight is determined by Faster-Rcnn, which extracts more information than the traditional fixed feature size encoder.

[0033] (2) Faster-Rcnn is an object detection model designed to identify object instances belonging to specific categories and locate them using bounding boxes. It can quickly locate one or more defects in the image, as well as the corresponding label and probability of belonging.

[0034] (3) In order to obtain multi-scale information, three gray images collected from the left 45 degrees, directly above, and right 45 degrees are used to correspond to the three-channel information of the initial input image of the encoder.

[0035] (4) Compared with the traditional convolutional neural network (cnn) image modeling and Vision transformer (ViT), the present application uses the faster-rcnn target detection method to extract the target frame in the encoding stage. For each picture, the target frame is first selected, and a closer relationship is established between the latest developments in visual and language tasks and target detection. The accuracy of the output sentence can be greatly improved compared with the traditional method. BRIEF DESCRIPTION OF DRAWINGS

[0036] Figure 1 Flowchart of the semiconductor defect analysis method of the present application for fast positioning and joint semantic interpretation;

[0037] Figure 2 Image sample example;

[0038] Figure 3 Defect image encoding network (Faster R-CNN) specific flowchart;

[0039] Figure 4 Image detection network Faster R-CNN output target frame result example graph;

[0040] Figure 5 Attention on LSTM module graph. DETAILED DESCRIPTION

[0041] The present application will be further described and explained in conjunction with the specific embodiments. The embodiments are only exemplary and do not circumscribe the scope of the disclosure. The technical features of each embodiment in the present application can be combined accordingly without conflict.

[0042] As Figure 1 and Figure 2 The semiconductor defect analysis method of rapid positioning combined with semantic interpretation provided by the present application mainly includes the following steps:

[0043] Obtain a wafer defect image diagnosis data set, including wafer defect SEM images and diagnosis reports;

[0044] The wafer defect image diagnosis data set used in this embodiment is produced from a 55nm real wafer production line, including 1475 defects, 4425 high-resolution gray-scale images of the same position wafer defect taken from the left, right and directly above, 7375 original sentences, covering 7 common defect categories in the chip manufacturing process. It is the first data set constructed in the aspect of wafer defect caption processing. The present application collects multiple scenes on different background process layers for each defect of the same category, so that the wafer defect image diagnosis data set has stronger richness and scalability. The wafer defect image diagnosis data set gives a comprehensive description of five directions for each defect, and the diagnosis report content involved includes the position, shape, size, concave-convex, deformation degree, process layer where the defect is located and further description of the background. The spatial resolution of the defect is distributed in the range of about 0.1 to 1um / pixel.

[0045] The following instructions are followed when constructing the image description sentences of the diagnosis report:

[0046] (1) A sentence contains more than six words, except for the fifth sentence used to define the image defect category.

[0047] (2) Describe all visible defects of the input defect image, one or more.

[0048] (3) Do not use ambiguous words and grammar for description.

[0049] (4) The first sentence describes the position, shape and size of the defect.

[0050] (5) The second sentence describes the concave-convex and deformation degree of the defect.

[0051] (6) The third sentence describes the process layer where the defect is located.

[0052] (7) The fourth sentence describes the background and composition information of the defect.

[0053] (8) Encourage the use of more rich vocabulary and flexible sentence patterns.

[0054] Figure 2are part of image sample examples, each of which contains a wafer defect SEM image, in addition to containing five sentences of diagnosis report, and the original file is named as SSID.token.txt.

[0055] For example, in the lower left corner of an image, its diagnosis report contains five sentences as follows: Figure 2

[0056] 1) An irregular elliptical defect is in the center position.

[0057] 2) This defect is protruding and severely deformed.

[0058] 3) This defect occurs in the contact layer tungsten deposition-chemical mechanical polishing process step.

[0059] 4) The background presents dark and is arranged in many small blocks.

[0060] 5) This is a pit defect.

[0061] In terms of data set processing, the original description SSID.token.txt of the data set (image ID address # title number + space + description) is traversed by five cycles, each sentence is converted into a list containing multiple words, and the result is saved in a new dictionary. Among them, the key is the picture id in the original dictionary, and the value is the processed word list. The finally generated wordmap.json is used for training and testing in the subsequent model together with the two kinds of json format files recording the captions and caplens information of the training and validation set, and the hdf5 format file recording the images information.

[0062] The defect image diagnosis network is constructed, and the defect image diagnosis network comprises a Faster R-CNN encoder and a decoder, and the decoder adopts an attention mechanism based LSTM; the defect image diagnosis network is trained by using a wafer defect image diagnosis data set, a wafer defect SEM image is input into the Faster R-CNN encoder to obtain a feature vector of the wafer defect SEM image, the feature vector and the last output of the decoder are input into the decoder to obtain a word vector representation of a word obtained by decoding this time; the word vector representations of all words generated by the decoder are sequentially spliced to form a sentence of a defect diagnosis task; and the sentences of different defect diagnosis tasks are combined to obtain a diagnosis report.

[0063] In the training process of the defect image diagnosis network, the sentences of the defect diagnosis task are used as the input of the LSTM decoder to guide the LSTM decoder to generate a target sentence.

[0064] The trained defect image diagnosis network is used to analyze a wafer defect SEM image to be processed to generate a diagnosis report.​

[0065] Compared with traditional CNN image modeling, the application extracts image regions based on a bottom-up mechanism (based on Faster R-CNN), each region is encoded as a relevant feature vector, and the feature weight is determined by a top-down mechanism. Since Faster R-CNN is a two-stage detector, it needs to perform region selection twice during processing. The first stage is called the region proposal network (RPN), which is a small network that slides over the intermediate feature layers of the CNN, predicting object scores and bounding box adjustments for each spatial location, these bounding boxes are based on predefined anchor boxes. Using the non-maximum suppression (NMS) technique, the best candidate regions are selected by crossing the intersection over union (IoU) threshold, these regions will be used as input for the second stage. In the second stage, the region of interest (RoI) pooling technique is applied to each candidate region to extract fixed-size feature maps, which are then sent to the last few layers of the CNN for further processing. The final output of the model includes the softmax distribution over class labels and class-specific bounding box refinements for each box proposal.

[0066] The application combines Faster R-CNN with ResNet-101. ResNet-101 is a deep residual network that can alleviate the gradient vanishing problem through skip connections, improving the training efficiency of deep networks. By combining Faster R-CNN with ResNet-101, the model can take advantage of the powerful feature extraction capability of ResNet-101 to further improve the accuracy of target detection. For each selected region, its mean-pooled convolutional features are extracted, with a feature vector dimension of 2048, providing rich bottom-up feature representations for the image. Faster R-CNN acts as a "hard" attention mechanism in the application, which selectively focuses on key regions in the image to filter out the most representative bounding box features from a large number of possible candidate regions. This mechanism effectively reduces the computational burden while retaining the most important visual information for image description. The specific method flow is shown in Figure 3 which details the complete process from image input to feature extraction, region selection, and feature encoding.

[0067] As Figure 3As shown, the process of the encoder is as follows: the image is input into the encoder, and the encoder part of the network adopts the Faster R-CNN structure. For a given input image, first, the Faster R-CNN uses a pre-trained convolutional neural network (ResNet101) as the backbone network to forward propagate the input image and extract feature maps (FeatureMap); wherein the pre-trained ResNet-101 is obtained by pre-training ResNet-101 on an ImageNet dataset. These feature maps capture multi-scale and multi-level semantic information of the image. Specifically, when a given input image (for example, with a size of 224x224x3) is given, the lower blocks of ResNet are first used to perform convolutional and pooling operations on the image until the C4 block. In this way, a 14x14x1024 feature map is generated, which contains high-level semantic information of the input image. As shown in the figure, on the feature map, a region proposal network (RPN) generates multiple regions of interest (Region of Interests, RoIs) through sliding windows. The RPN uses the feature map extracted by the encoder to predict the position and target class of each sliding window. This includes using multiple anchor boxes of different scales and shapes to cover each position of the feature map. The RPN predicts two outputs for each anchor box, one is a binary classification problem (foreground or background), and the other is a bounding box regression problem (predicting the offset of the box). By setting a threshold and non-maximum suppression (Non-Maximum Suppression, NMS), high-quality candidate regions are selected. Figure 4

[0068] For each candidate region generated by the RPN, the Faster R-CNN uses a RoI pooling layer to convert it into a fixed-size feature. The RoI pooling layer maps candidate regions of different sizes to feature vectors with the same dimensions, usually a WxH feature map, where W is the width and H is the height, and W and H are fixed values (i.e., pre-set). Specifically, feature calculation is performed separately for each region, and the C5 block of ResNet-101 is applied to the features after the 14x14 RoIPool operation. The output of C5 is then averaged pooled by AvgPool into the final vector for each region as the bottom-up feature. Since all final region features come from C5, the detector can be converted back to the ResNet classifier, and the same layer C5 is used as the output feature of the present application. After RoI pooling, the feature map of each candidate region is converted into a fixed-length vector, which is then sent to the subsequent decoder for further feature extraction.

[0069] ​In the present application, the Faster R-CNN encoder first uses a pre-trained ResNet-101 to extract features from the wafer defect SEM image, obtains a feature map and inputs it into the RPN, obtains a plurality of candidate regions, then inputs all the candidate regions into the RoI pooling layer to obtain a fixed-length feature vector, inputs the fixed-length feature vector into the pre-trained ResNet-101 for feature extraction, and then performs average pooling operation on the feature vector after feature extraction to obtain a final feature vector and output it into the LSTM decoder. The key to the whole process is RPN and RoI pooling, which enables Faster R-CNN to effectively extract candidate target regions from the original image and convert them into fixed-size features suitable for classification and bounding box prediction.

[0070] It is worth noting that in this process, in order to pre-train the bottom-up attention model, the present application needs to first initialize the Faster R-CNN with the pre-trained ResNet-101. Then train on the data set of the present application.

[0071] Due to the Faster R-CNN bottom-up feature extraction method adopted by the present application, it is necessary to download previously pre-processed feature data, and an additional folder is prepared to store the position and size of the target box. The features that need to be extracted include: bbox is the bounding box information, num_bbox is the number of bounding boxes, image_h is the height of the picture, image_w is the width of the picture, and info is other important information.

[0072] In the decoding stage, referring to the paper "Show, Attend and Tell", the present application inputs the defect SEM image features obtained by the feature extraction network and the corresponding sentence into the LSTM decoder (i.e. the LSTM decoder based on attention mechanism for different defect diagnosis tasks), and trains the model to predict words one by one for a given image. The network will generate a word at each time step according to the context vector, the previous hidden state and the previously generated word, so as to generate a complete sentence, and the next step of the network depends on the sequence of words that have been generated. The initial memory state and hidden state of the LSTM are predicted by the average of the initial vectors provided by two independent mlps.

[0073] In sequence generation tasks that employ LSTM (Long Short-Term Memory) as the decoder based on attention mechanism, the attention mechanism plays a crucial role. This mechanism enables the decoder to flexibly focus on the most relevant parts of the input data at each step of the generation process. Specifically, in the context of image description generation, the decoder first receives image feature representations from the encoder, which capture the visual information of the image. Subsequently, the decoder combines these features with the given linguistic context, such as the user's query or previous descriptions, as well as the previously generated output, to decide the next word in the sequence. In this way, the decoder can generate descriptions that closely relate to the image content while maintaining coherence with the previous descriptions.

[0074] The core of the attention mechanism lies in its ability to dynamically assign different weights to the decoder's hidden states, which are adjusted based on the relevance of the current generation task. For example, when generating a description of a specific object, the decoder can use attention scores to make the model pay more attention to the features of that object in the image. With each word generated, the decoder updates its hidden state while adjusting the attention distribution to reflect the influence of the newly generated word on the context.

[0075] As shown in Figure 5 , the decoder part of the defect image diagnosis network uses the LSTM attention mechanism, which utilizes an LSTM decoder to step-by-step decode the feature vectors to generate word vector representations of the words corresponding to the respective defect diagnosis task, generates words based on the word vector representations, and concatenates all the words to form a sentence for the defect diagnosis task; the sentences for different defect diagnosis tasks are combined to obtain a diagnosis report.

[0076] During the step-by-step decoding process of the LSTM decoder on the feature vectors, the LSTM decoder interacts with the feature vectors, the previous decoding result of the decoder, and the sentence for the defect diagnosis task to obtain the word vector representation of the next word. The entire process is iterative until the generated sequence reaches a predetermined length or encounters a sequence end marker (such as a specific end symbol). That is, during the step-by-step decoding process of the LSTM decoder on the feature vectors, the previous decoding result is used as the input for the next decoding step, and the previous decoding result at the initial time is empty. In this way, through the careful design of LSTM and attention mechanism, the model can gradually construct an accurate, fluent, and content-rich subtitle sequence, effectively integrating visual and linguistic information to achieve high-quality description of image content.

[0077] Specifically, first, the image features extracted by the encoder are used as the initial context for the decoder. These features capture the visual information of the image, providing a foundation for generating the description. Next, the LSTM decoder generates a context vector at each time step, which is part of the decoder's hidden state and contains information from all previous time steps, including knowledge learned from the image features and previously generated language information. At each time step, the decoder computes attention weights over the image features. These weights are derived through the interaction of the decoder's current hidden state and the image features, implemented by a learnable attention score function. Using the computed attention weights, the image features are weighted to generate a weighted image feature representation that highlights the most relevant parts for the current generation task.

[0078] The weighted image features are combined with the decoder's context vector to form the input to the decoder. This fusion process can be achieved through simple concatenation, weighted averaging, or other more complex fusion techniques. The LSTM decoder updates its hidden state based on the fused input and generates a word vector representation for the next word. During this process, the decoder's hidden state continuously accumulates the interaction of language information and visual information. Finally, the word vector representation generated by the decoder is used to determine the next most likely word through a softmax layer, which is then added to the sequence of the description being generated. The newly generated word is fed back into the decoder as part of the input for the next step of generation. In this way, the decoder takes into account the previously generated words when generating each new word, achieving the accumulation and interaction of language information.

[0079] For the training of the defect image diagnosis network, a cross-entropy loss function is used to calculate the loss value of the defect image diagnosis network, and the trained defect image diagnosis network is obtained through iterative optimization. The cross-entropy loss function is used to optimize the difference between the generated subtitles and the real subtitles. Specifically, the cross-entropy loss function first converts the predicted output of each word (usually a continuous word vector representation) into a probability distribution through a softmax layer, which generates a probability value between 0 and 1 for each possible word, and the sum of these probability values is 1. Then, the loss function calculates the gap between this probability distribution and the actual distribution of words in the real subtitles. Each word in the real subtitle sequence is represented by one-hot encoding, i.e., the correct word position is 1 and the other positions are 0.

[0080] The smaller the value of the loss function, the higher the consistency between the predicted subtitles and the real subtitles. During the training process, the goal of the present application is to adjust the parameters of the model through the backpropagation algorithm Adam optimizer to minimize the value of the loss function.

[0081] In the present application, the testing of the defect image diagnosis network is to select the best caption sequence by maximizing the generation probability, and output the generated image caption. Specifically, during the testing, the present application adopts Beam Search to maximize the generation probability of the entire caption sequence. Multiple possible word combinations are considered, and the sequence with the highest probability is selected. With the addition of each new word, the context of the decoder is updated to reflect the influence of the newly generated word on the caption sequence. This process continues until the end-of-sequence marker is generated or the predetermined maximum length is reached.

[0082] The generated caption sequence is evaluated for its quality by BLEU, ROUGE, etc. to guide the further optimization of the model. Finally, the generated caption sequence is displayed together with the image, and the image attention activation map corresponding to each word of the image. Not only does it enhance the intelligibility of the image, but it also provides a practical and intuitive way for the explainability of the model.

[0083] The present embodiment also provides a semiconductor defect analysis system for rapid positioning of joint semantic explanation, comprising:

[0084] a data acquisition module configured to acquire a wafer defect image diagnosis dataset in a training phase, the wafer defect image diagnosis dataset comprising wafer defect SEM images and diagnosis reports, wherein the diagnosis reports contain multiple sentences for different defect diagnosis tasks; and acquire a wafer defect SEM image to be analyzed in a prediction phase;

[0085] a defect image diagnosis network module comprising a Faster R-CNN encoder and an LSTM decoder based on an attention mechanism for different defect diagnosis tasks; the wafer defect SEM image is input into the Faster R-CNN encoder to obtain a feature vector of the wafer defect SEM image, the LSTM decoder is used to decode the feature vector step by step to generate a word vector representation of the word corresponding to the corresponding defect diagnosis task, the word is generated according to the word vector representation, and all the words are sequentially spliced to form a sentence for the defect diagnosis task; the sentences for different defect diagnosis tasks are combined to obtain a diagnosis report;

[0086] a network training module configured to train the defect image diagnosis network using the wafer defect image diagnosis dataset, wherein the sentences for the defect diagnosis task are input into the LSTM decoder during the training process of the defect image diagnosis network to guide the LSTM decoder to generate target sentences;

[0087] a defect analysis module configured to analyze the wafer defect SEM image to be processed using the trained defect image diagnosis network to generate a diagnosis report.

[0088] For the system embodiment, since it basically corresponds to the method embodiment, the relevant part is described in the method embodiment, and the implementation method of the remaining modules is not described here. The system embodiment described above is only illustrative, wherein the units described as separate components can or can not be physically separated, and the components displayed as units can or can not be physical units, i.e., they can be located in one place or distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the present application according to actual needs. Those skilled in the art can understand and implement it without creative labor.

[0089] The system embodiment of the present application can be applied to any device with data processing capability, which can be a device or apparatus such as a computer. The system embodiment can be implemented by software, hardware, or a combination of software and hardware. Taking software implementation as an example, as a logical device, it is formed by reading the corresponding computer program instructions in the non-volatile memory into the memory and running by the processor of the device with data processing capability.

[0090] Obviously, the above-described embodiments and drawings are only some examples of the present application, and those skilled in the art can also apply the present application to other similar situations according to the drawings without creative labor. In addition, it can be understood that although the work done in the development process can be complex and long, some design, manufacture or production changes according to the technical content disclosed in the present application are only routine technical means for those skilled in the art, and should not be regarded as insufficient disclosure of the present application. Without departing from the concept of the present application, several modifications and improvements can also be made, which are within the scope of protection of the present application. Therefore, the scope of protection of the present application should be subject to the appended claims.

Claims

1. A method for rapidly locating semiconductor defect analysis based on joint semantic interpretation, characterized in that, include: Obtain a wafer defect image diagnostic dataset containing SEM images of wafer defects and diagnostic reports, wherein the diagnostic reports contain multiple sentences for different defect diagnostic tasks; A defect image diagnosis network is constructed, comprising a Faster R-CNN encoder and an attention-based LSTM decoder for different defect diagnosis tasks. The network is trained using a wafer defect image diagnosis dataset. A wafer defect SEM image is input into the Faster R-CNN encoder to obtain its feature vector. The LSTM decoder then progressively decodes the feature vector to generate word vector representations corresponding to the respective defect diagnosis tasks. Words are generated based on these word vector representations, and all words are sequentially concatenated to form sentences for the defect diagnosis tasks. The combination of sentences from different defect diagnosis tasks yields a diagnostic report. During the training of the defect image diagnosis network, the sentence of the defect diagnosis task is used as the input of the LSTM decoder to guide the LSTM decoder to generate the target sentence. The trained defect image diagnostic network is used to analyze the SEM images of wafer defects to be processed and generate a diagnostic report.

2. The semiconductor defect analysis method for rapid localization and joint semantic interpretation according to claim 1, characterized in that, The defect diagnosis task includes: Defect location, shape, and size diagnostic task; The task of diagnosing the concavity / convexity and the degree of deformation of defects; The task of diagnosing the process layer where the defect is located; The background and composition of defects are information required for diagnostic tasks; Defect category diagnosis task; The defect diagnosis task includes at least one of the above-mentioned tasks.

3. The semiconductor defect analysis method for rapid localization and joint semantic interpretation according to claim 2, characterized in that, The sentences in the defect diagnosis task in the diagnostic report simultaneously meet the following conditions: In addition to the statements corresponding to defect category diagnosis tasks, a statement should contain at least six words; Do not use ambiguous words and grammar; The statement contains all the defects.

4. The semiconductor defect analysis method for rapid localization and joint semantic interpretation according to claim 1, characterized in that, The wafer defect SEM images in the wafer defect image diagnostic dataset are obtained by synthesizing wafer defect SEM images taken from different angles.

5. The semiconductor defect analysis method for rapid localization and joint semantic interpretation according to claim 1, characterized in that, Before training the defect image diagnosis network, sentences in the diagnostic reports of the wafer defect image diagnosis dataset are preprocessed. The preprocessing includes: Convert all uppercase letters in the sentence to lowercase, remove all punctuation marks in the sentence and perform word segmentation, convert each sentence into a list containing multiple words, and store the processing results in a dictionary where the key is the ID of the wafer defect SEM image and the value is a list containing multiple words. All sentences in the diagnostic report are subjected to the same preprocessing.

6. The semiconductor defect analysis method for rapid localization and joint semantic interpretation according to claim 1, characterized in that, The Faster R-CNN encoder first uses a pre-trained ResNet-101 to extract features from the wafer defect SEM image, obtains a feature map, and inputs it into the RPN to obtain multiple candidate regions. Then, all candidate regions are input into the RoI pooling layer to obtain a fixed-length feature vector. The fixed-length feature vector is then input into the pre-trained ResNet-101 for feature extraction. The extracted feature vector is then subjected to average pooling to obtain the final feature vector, which is then output to the LSTM decoder.

7. The semiconductor defect analysis method for rapid localization and joint semantic interpretation according to claim 6, characterized in that, The pre-trained ResNet-101 was obtained by pre-training ResNet-101 on the ImageNet dataset; and before training the defect image diagnosis network, the Faster R-CNN encoder was initialized using the pre-trained ResNet-101.

8. The semiconductor defect analysis method for rapid localization and joint semantic interpretation according to claim 1, characterized in that, During the stepwise decoding process of the feature vector by the LSTM decoder, the decoding result of the previous step is used as the input for the next step. Initially, the decoding result of the previous step is empty.

9. The semiconductor defect analysis method for rapid localization and joint semantic interpretation according to claim 1, characterized in that, The loss value of the defect image diagnosis network is calculated using the cross-entropy loss function, and the trained defect image diagnosis network is obtained through iterative optimization.

10. A semiconductor defect analysis system for rapid localization and joint semantic interpretation, characterized in that, include: The data acquisition module is used to acquire a wafer defect image diagnostic dataset during the training phase, which includes wafer defect SEM images and diagnostic reports, wherein the diagnostic reports contain multiple sentences for different defect diagnostic tasks; and to acquire wafer defect SEM images to be analyzed during the prediction phase. The defect image diagnosis network module includes a Faster R-CNN encoder and an attention-based LSTM decoder for different defect diagnosis tasks. A wafer defect SEM image is input into the Faster R-CNN encoder to obtain the feature vector of the wafer defect SEM image. The LSTM decoder then progressively decodes the feature vector to generate word vector representations corresponding to the words in the respective defect diagnosis task. Words are generated based on the word vector representations, and all words are sequentially concatenated to form a sentence for the defect diagnosis task. The combination of sentences from different defect diagnosis tasks yields a diagnostic report. The network training module is used to train a defect image diagnosis network using a wafer defect image diagnosis dataset. During the training process of the defect image diagnosis network, the sentence of the defect diagnosis task needs to be used as the input of the LSTM decoder to guide the LSTM decoder to generate the target sentence. The defect analysis module is used to analyze the SEM images of wafer defects to be processed using a trained defect image diagnostic network and generate a diagnostic report.

Images