A method and system for automatically matching wafer reference layout and SEM image

Abstract

The present invention discloses an automatic matching method and system for a wafer reference layout and a SEM image, and belongs to the field of image matching technology. The SEM image and the reference layout image of the wafer pattern are obtained and a style transfer model from the SEM to the layout style image is trained, including a cyclic generative adversarial network, a wafer feature pattern segmentation branch network and a wafer pattern line edge detection branch network. The number of residual blocks in the generator for generating SEM and layout style images in the cyclic generative adversarial network is proportional to the image entropy of the SEM and reference layout images; the cyclic generative adversarial network loss, the position consistency loss based on the center of gravity distribution map, and the edge retention loss based on the contour information are designed in the training stage; the SEM image to be matched is processed into a layout style image by using the trained style transfer model before completing the matching process. The present invention solves the matching problem of the wafer reference layout and the SEM image.

Description

Technical Field

[0001] The invention belongs to the technical field of image matching, and in particular relates to an automatic matching method and system for a reference layout and a SEM image. Background Art

[0002] As the size of semiconductor devices continues to shrink, the manufacturing process becomes more and more complex, and the semiconductor chip process has evolved from 65 nanometers to 7 nanometers, 5 nanometers, and 3 nanometers. This places higher demands on the patterning technology of wafers, such as self-aligned double patterning, self-aligned quadruple patterning, and multiple photolithography etching steps have become more common. However, most traditional lithography systems use ArF lithography machines with a wavelength of 193 nanometers, which is significantly larger than the size of the current node. The difference between the wavelength of the light source of this lithography machine and the size of the manufacturing node may cause manufacturing defects, also known as lithography hotspots. These defects are divided into random defects and systematic defects. Random defects can be caused by various reasons, such as tiny particles caused by dust adhering to the surface of the wafer or mechanical damage that may occur on the surface of the wafer after slicing or CMP process. Systematic defects are caused by equipment, materials, and environmental problems in the process, or because the design pattern of the layout does not match the actual process window. According to the severity of the defect, it is divided into hard defects and soft defects. Hard defects can have a significant impact on semiconductor performance, while soft defects bring potential risks to the performance of the device. All defects must be fully inspected and identified by dedicated equipment. Review SEM (Scanning Electron Microscope, referred to as ReviewSEM) is widely used to automatically inspect SEM images, and is used for defect location and automatic classification. Review SEM mainly implements defect detection based on two methods: Die to Die (D2D) inspection and Die to Database (D2DB) inspection. D2D inspection detects hot spots by comparing SEM images with reference SEM images, and classifies them according to the rules stored in the database. Its main disadvantage is that a reference SEM image needs to be selected for each layout pattern, which is difficult to automate. In this context, the layout has become an extremely important source of information because it contains key information such as layout pattern, size and structure. Therefore, in the chip process of smaller nodes, Review SEM combines SEM images with reference layouts to find defects, which is also called D2DB inspection. Using layout information can not only greatly shorten the analysis time and the inherent defect location error in SEM images, but also can be used in the image comparison process of optical mask production (OPC). Therefore, aligning the reference layout with the corresponding scanning electron microscope (SEM) image becomes one of the key tasks for D2DB inspection in the semiconductor industry.

[0003] In traditional schemes, the contours of SEM images are usually extracted first, such as using the Canny algorithm, and with the help of scaling, relaxation, differential filters and averaging techniques, and then matched with the corresponding target pattern. However, due to the diversity and irregularity of patterns and defects, it is difficult to accurately extract contours from noisy SEM images, and this method is often limited in practice. In addition, the image analysis becomes more complicated because the SEM image itself has some inherent characteristics, including missing edge information, rounded corners, line edge roughness, similar pixel intensity between background and pattern, and uneven particle texture. Therefore, the review process inevitably requires inspection by experienced experts, which is laborious, error-prone and time-consuming. It should be clear that the accuracy of wafer patterns is of vital importance in semiconductor manufacturing. The nano-level process requires extremely high accuracy of wafer patterns. Even small deviations may lead to failures, losses, and significant reductions in device performance or even failure on the production line.

[0004] With the introduction of deep learning, object detection has become one of the most successful technologies in computer vision and serves as the basis for various applications, including anomaly detection in medical images, autonomous driving systems, and pose estimation. There are many successful object detection models. On the one hand, it is not easy to achieve accurate alignment between SEM images and reference layouts, because SEM images may have different pattern shapes and styles compared to reference layouts, which makes accurate alignment difficult. Although it is a feasible method to transfer the style of SEM images to reference layouts and then match them, the classic image style transfer method focuses more on the conversion of visual style and appearance between different images, and pays less attention to the accuracy of shape and structure, such as the generation from real-life style to comic style, from landscape to Van Gogh style, or from sketch to color image. These transfers focus more on the authenticity of human perception. In this case, it is not advisable to simply transfer style and ignore pattern accuracy. On the other hand, most object detection model training requires a large amount of paired image data. Since SEM images usually require high resolution and multi-angle scanning to capture tiny structures and details on the wafer, and layout images require high-precision measurement and modeling of the wafer, the acquisition of paired data requires a lot of expertise and technical support, as well as a large amount of experimental data and verification work. For D2DB tasks in the semiconductor manufacturing field, it is expensive and difficult to obtain and label this data. Summary of the invention

[0005] In view of the above problems, the present invention proposes an automatic matching method and system for wafer reference layout and SEM image. The present invention is implemented based on an improved CycleGAN model, does not require paired data, and combines prior knowledge such as image entropy of SEM and reference layout images to design the network depth of two generators in the cyclic generative adversarial network, so that it can better adapt to the unique complexity mismatch problem between wafer layout and SEM pattern; at the same time, an attention mechanism is introduced to make the network pay more attention to the characteristics of the wafer pattern, thereby improving the accuracy of pattern generation. In view of the characteristics of the SEM and layout image domains, the present invention optimizes the loss function of the model: introduces a pre-trained wafer feature pattern segmentation network and a wafer pattern line edge detection network, which correspond to the position consistency loss and edge retention loss of the wafer pattern respectively, ensuring the pattern position invariance and contour similarity on the migrated wafer, which is conducive to the high-precision migration of the wafer pattern. The present invention solves the matching problem of wafer reference layout and SEM image, and provides an effective solution for D2DB tasks in the field of semiconductor manufacturing.

[0006] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0007] In a first aspect, the present invention provides a method for automatically matching a wafer reference layout and a SEM image, comprising the following steps:

[0008] Step 1, obtaining a SEM image of a wafer pattern and a reference layout image dataset, wherein the SEM image and the reference layout image do not need to be paired;

[0009] Step 2, constructing a style transfer model from SEM to layout style image, including a cyclic generative adversarial network, a wafer feature pattern segmentation branch network and a pattern line edge detection branch network, wherein the cyclic generative adversarial network includes two generators and two discriminators, and the generator adopts a network structure based on residual blocks, and the number of residual blocks in the generator for generating SEM style image and the generator for generating layout style image is proportional to the image entropy of SEM image and reference layout image;

[0010] The wafer feature pattern segmentation branch network is used to segment the real image and the generated image, label the segmented images in sequence and generate corresponding center of gravity distribution maps; the wafer pattern line edge detection branch network is used to perform wafer pattern line edge detection on the real image and the generated image to generate contour information;

[0011] Step 3, using the image dataset obtained in step 1 to train the style transfer model, the loss functions used in the training phase include cyclic generative adversarial network loss, position consistency loss based on centroid distribution map, and edge preservation loss based on contour information;

[0012] Step 4: Use the trained style transfer model to process the SEM image to be matched into a layout-like style image, match the layout-like style image with the reference layout image to be matched, process the SEM image according to the matching information and locate it to the corresponding reference layout position, and complete the automatic matching process between the SEM image and the reference layout.

[0013] Furthermore, the generator includes an initialized convolution layer, a downsampling layer, several residual blocks, an upsampling layer and an output layer connected in sequence, and the upsampling layer and the downsampling layer both include several convolution layers and an attention module located after the last convolution layer.

[0014] Furthermore, the discriminator includes a plurality of convolution blocks, a plurality of attention modules, and an output layer. The attention module is located between two adjacent convolution blocks, and the convolution block is composed of a plurality of convolution layers.

[0015] Furthermore, before the SEM image is input into the style transfer model, a step of preprocessing the SEM image is also included, wherein the preprocessing includes denoising and high-resolution processing.

[0016] Furthermore, the wafer feature pattern segmentation branch network adopts a pre-trained TernausNet network.

[0017] Furthermore, the wafer pattern line edge detection branch network adopts a pre-trained U-Net network, and the labels used in the pre-training process are contour information extracted after performing wafer pattern line edge detection on SEM images and reference layout images through the Canny algorithm.

[0018] Furthermore, the cycle generative adversarial network loss includes adversarial loss, cycle consistency loss and identity mapping loss.

[0019] Furthermore, the formulas for the position consistency loss based on the centroid distribution map and the edge preservation loss based on the contour information are as follows:

[0020]

[0021]

[0022] in, are position consistency loss and edge preservation loss respectively, SEM and GDS are the real SEM image and reference layout image respectively, G SEM , G GDS are generators for generating SEM style images and layout style images, respectively. c (.) is the centroid distribution diagram of the wafer feature pattern segmentation branch network after segmentation, ‖.‖1 is the L1 norm, E d(.) is the contour information generated by the wafer pattern line edge detection branch network, E SEM , GDS They represent the summation or integration of all samples sampled from the distribution of the SEM image and GDS image domains, respectively, to calculate the expected value of the function on these samples.

[0023] Furthermore, the step 4 is specifically as follows:

[0024] The SEM image to be matched is processed into the input size of the trained style transfer model, and a layout-like style image is generated by using a generator for generating layout-style images; the SIFT algorithm is used to extract key feature points of the generated layout-like style image and the reference layout to be matched, and the key features are matched using a FLANN matcher to obtain the geometric transformation relationship between the layout-like style image and the reference layout; based on the geometric transformation relationship, the position coordinates and scaling factor of the layout-like style image in the reference layout are obtained; finally, the SEM image is processed according to the position coordinates and scaling factor and positioned at the corresponding reference layout position, thereby completing the automatic matching process between the SEM image and the reference layout.

[0025] In a second aspect, the present invention proposes an automatic matching system for a wafer reference layout and a SEM image, which is used to implement the above-mentioned automatic matching method for a wafer reference layout and a SEM image.

[0026] The beneficial effects of the present invention are:

[0027] (1) High degree of automation: In the data set collection stage, there is no need to collect paired SEM and layout data, which greatly reduces the difficulty of data collection. At the same time, this model can realize end-to-end training and matching process without manual intervention, which improves the degree of automation and reduces the impact of human errors and subjective factors.

[0028] (2) Designing asymmetric SEM and layout style image generators based on prior knowledge of image complexity: The model proposed in the present invention combines prior knowledge such as image entropy of SEM and layout images, and optimizes the network depth of the two generators in the recurrent generative adversarial network, so that it can better adapt to the unique complexity mismatch problem between wafer layout and SEM pattern.

[0029] (3) Introducing the attention mechanism: Inserting the attention mechanism module into the generator and the discriminator makes the network pay more attention to the characteristics of the wafer pattern, thereby improving the accuracy of pattern generation.

[0030] (4) Targeted optimization of the loss function based on the characteristics of SEM and layout data: a pre-trained wafer feature pattern segmentation network and a wafer pattern line edge detection network were introduced, corresponding to the position consistency loss and edge preservation loss of the wafer pattern, respectively, to ensure the pattern position invariance and contour similarity on the migrated wafer, which is conducive to the high-precision migration of the wafer pattern. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 Schematic diagram for automatic matching of reference layout and SEM image.

[0032] Figure 2 This is the generator structure of the cyclic generative adversarial network in the present invention.

[0033] Figure 3 This is an implementation flowchart of SEM style migration to layout-like style based on the style migration model.

[0034] Figure 4 It is a flow chart of the automatic matching method of the reference layout and the SEM image proposed by the present invention. DETAILED DESCRIPTION

[0035] The following description is used to disclose the present invention so that those skilled in the art can implement the present invention.

[0036] The accompanying drawings are only schematic illustrations of the present invention and are not necessarily drawn to scale. Some of the block diagrams shown in the accompanying drawings are functional entities that do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0037] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the steps. For example, some steps may be decomposed, while some steps may be combined or partially combined, so the actual execution order may change according to the actual situation.

[0038] The present invention proposes an automatic matching method for wafer reference layout and SEM image, which is implemented based on an improved CycleGAN model. The principle is as follows: Figure 1 As shown, for the SEM image (dark image) and the reference layout image (light image) to be matched, the SEM image is first style-transferred to generate a layout-like image of the same size as the SEM image, and then the layout-like image is matched with the reference layout image to obtain a matching relationship. The matching of the SEM image and the reference layout image can be directly completed based on the matching relationship.

[0039] Specifically, if Figure 4As shown, the matching method mainly includes the following steps:

[0040] S1, collects scanning electron microscope (SEM) image dataset and reference layout dataset.

[0041] In this embodiment, the required image dataset comes from a real industrial dataset, including 3,000 SEM images and reference layout images, and the sizes of the reference layout images and SEM images are 512×512×3 and 256×256×3, respectively. In order to match the input requirements of the model, the reference layout images are randomly cropped to a size of 256×256×3, and 2,100 images are selected from the cropped reference layout image dataset and SEM image dataset as training sets, and the remaining 900 SEM images and 900 reference layout images are used as validation sets.

[0042] S2, preprocessing the SEM image.

[0043] There may be various complex noises, illumination changes and interferences in SEM images and layouts, and traditional contour extraction and pattern matching are often difficult to handle these complex situations. The present invention first performs unsupervised denoising and high-resolution processing of SEM images using BM3D and ZSSR, thereby improving the ability to process complex images.

[0044] First, the BM3D (Block Matching 3D) algorithm is used to reduce the noise of SEM images. The BM3D algorithm is based on the principle of local similarity of images, decomposes the image into local blocks, and uses the information of similar blocks for denoising. Specifically, the BM3D algorithm first searches for similar blocks in the SEM image, and uses these similar blocks for block matching and mean filtering, and finally uses nonlinear transformation to further reduce the noise. This algorithm can effectively reduce Gaussian noise and other types of noise in SEM images, thereby improving the quality of SEM images.

[0045] Next, the ZSSR (Zero-Shot Super-Resolution) model is used to perform high-resolution processing on the denoised SEM images. The ZSSR model can convert low-resolution images into high-resolution images without using reference images by learning the intrinsic structure and features of the image. Feature extraction and reconstruction are performed through a multi-layer convolutional neural network (CNN), combining the local information and global structure inside the image to achieve high-resolution reconstruction of the image. Compared with traditional super-resolution methods, the ZSSR model has better generalization ability and higher reconstruction quality, and is particularly suitable for processing low-resolution SEM images. After BM3D denoising and ZSSR high-resolution processing, the SEM image will obtain a clearer and higher-quality SEM image, providing a more reliable basis for subsequent image matching and analysis.

[0046] S3, builds a migration model from SEM to layout style images. The model includes a CycleGAN-based backbone network and two branch networks: a wafer feature pattern segmentation branch network and a wafer pattern line edge detection branch network.

[0047] like Figure 3 As shown, the migration model construction steps include:

[0048] Step 1: Build a backbone network based on the improved CycleGAN.

[0049] The backbone network structure based on CycleGAN includes two generators and two discriminators, which are used to generate images in the SEM domain and images in the layout (GDS) domain (Generator_SEM, Generator_GDS), and to discriminate whether they are real images in the SEM domain and images in the GDS domain (Discriminator_SEM, Discriminator_GDS). Through adversarial training, the goal of the generator is to deceive the discriminator as much as possible, so that the generated fake images are visually difficult to distinguish from real images, while the goal of the discriminator is to distinguish real images from fake images as accurately as possible. Through adversarial training, CycleGAN can realize image conversion between the two domains of SEM and GDS, thereby achieving the effect of style transfer.

[0050] The generator of the original CycleGAN network consists of five parts, including an initialization convolution block, downsampling of two layers of convolution, a residual block, upsampling of two layers of convolution, and an output layer. Since the SEM image is generated by electron beam scanning, it contains a large amount of electronic information, and the original format of the layout image is a GDS file, which uses a binary format. The generated layout image is a binary image and contains only two pixel values. Therefore, for Generator_SEM and Generator_GDS, the task complexity of generating images is not equal. Therefore, for the same network structure, the complexity of the image can be quantified according to the image entropy, so the present invention improves the generator structure of CycleGAN, and selects different numbers of residual blocks n_SEM and n_GDS for Generator_SEM and Generator_GDS according to the image entropy as the prior knowledge of network depth design. The larger the image entropy, the more residual blocks there are. In this embodiment, n_SEM and n_GDS are 18 and 9 respectively. At the same time, the attention module (CBAM) is inserted after the second part (downsampling of two layers of convolution) and the fourth part (upsampling of two layers of convolution) of Generator_SEM and Generator_GDS. The calculation logic of CBAM belongs to the prior art in this field and will not be repeated here. Figure 2As shown in the figure, the operation process of the improved generator is as follows: (1) Initialization stage: fill the input image with 7x7 borders, then output a 64-channel feature map through a 7x7 convolution operation, and perform instance normalization (IN) and ReLU activation function processing; (2) Downsampling stage: two layers of convolution + CBAM, each layer of convolution includes a 3x3 convolution operation and IN, ReLU processing, and outputs a 256-channel feature map; (3) Residual block stage: Generator_SEM and Generator_GDS are respectively After being processed by n_SEM and n_GDS residual blocks, each residual block contains two 3x3 convolution operations and IN processing, and LeakyReLU activation is performed at the same time; (4) Upsampling stage: including two layers of convolution layers + CBAM, each convolution layer includes upsampling operation, 3x3 convolution operation and IN processing, and outputs 64-channel feature map; (5) Output layer stage: through 7x7 boundary padding and 7x7 convolution operation, a feature map with the same number of channels as the input image is output, and the generated image is output through the Tanh activation function.

[0051] In addition, the discriminators Discriminator_SEM and Discriminator_GDS of the original CycleGAN network adopt the PatchGAN structure and also use convolution operations for feature extraction. First, feature extraction is performed through four convolution blocks, and finally through the convolution operation, each pixel corresponds to a binary value, indicating whether the local area corresponding to the pixel is real. The results of each area are averaged to obtain the discrimination result. The present invention inserts CBAM after the first three convolution blocks of Discriminator_SEM and Discriminator_GDS to help the network better capture the global features and overall structure of the input image.

[0052] Step 2: Construct a branch network for wafer feature pattern segmentation.

[0053] The wafer feature pattern segmentation branch network Ec adopts the TernausNet network, which is a U-Net architecture with a VGG11 encoder and is only used in the training phase. The U-Net structure is well-known for its excellent image segmentation performance. Its encoder-decoder structure and jump connection design enable it to effectively capture details in the image. The network weights pre-trained on the ImageNet large dataset are imported during the initialization process. The ImageNet dataset is a large-scale image database used to train and test large-scale image recognition systems. By using the network weights pre-trained on ImageNet, the previously learned common features can be utilized to accelerate the convergence of the model and improve the performance of the model. In order to better adapt to the characteristics of the SEM image and layout datasets, the pre-trained network weights are further fine-tuned so that the model can better learn and understand the specific features between the wafer reference layout and the SEM image, thereby improving accuracy and stability. The segmentation branch network segments the input image, labels the segmented images in sequence, and generates a centroid distribution map corresponding to the image. In this embodiment, during the training phase, Generator_SEM is used to generate a SEM domain image corresponding to the original GDS domain image, and the wafer feature pattern segmentation branch network is used to segment the original GDS domain image and the generated SEM domain image, respectively, and a centroid distribution map is generated; similarly, Generator_GDS is used to generate a GDS domain image corresponding to the original SEM domain image, and the wafer feature pattern segmentation branch network is used to segment the original SEM domain image and the generated GDS domain image, respectively, and a centroid distribution map is generated.

[0054] Step 3: Construct a wafer pattern line edge detection branch network.

[0055] The wafer pattern line edge detection branch network Ed uses a U-Net network pre-trained by the contour extracted based on the canny algorithm, which is only used in the training stage. The Canny edge detection algorithm is a classic edge detection method. In the present invention, the Canny algorithm is first used to detect the wafer pattern line edges of the SEM image and the layout image to extract accurate contour information. Then, these contour information are used as labels to pre-train the U-Net network to learn to accurately extract these contours from the input image, thereby obtaining the pre-trained wafer pattern line edge detection branch network. In this embodiment, in the training stage, Generator_SEM is used to generate the SEM domain image corresponding to the original GDS domain image, and the wafer pattern line edge detection branch network is used to extract contour information from the original GDS domain image and the generated SEM domain image respectively; similarly, Generator_GDS is used to generate the GDS domain image corresponding to the original SEM domain image, and the wafer pattern line edge detection branch network is used to extract contour information from the original SEM domain image and the generated GDS domain image respectively.

[0056] S4, design loss function.

[0057] The loss function is designed according to the task requirements: 1. Style transfer from SEM image to layout image; 2. Accurate matching of wafer pattern position before and after migration; 3. Ensuring edge similarity of wafer pattern before and after migration. Based on this, the loss function designed by the present invention includes 5 parts: the basic loss function of the two GAN networks (adversarial loss), cycle consistency loss function, identity mapping loss, position consistency loss function, and edge preservation loss function.

[0058] (1) Adversarial Loss:

[0059] The adversarial loss is applied to both mapping directions. For the mapping function G GDS :SEM→GDS and its discriminator D GDS , the adversarial loss is expressed as:

[0060]

[0061] Among them, G SEM , G GDS are generators for generating SEM domain images and GDS domain images, respectively. GDS is a discriminator used to identify GDS domain images. SEM and GDS are real SEM domain images and GDS domain images respectively. SEM , GDS Respectively represent the summation or integration of all samples sampled from the distribution of SEM image and GDS image domain, which are used to calculate the expected value of the function on these samples. GDSAims to generate fake images G that are completely indistinguishable from images in the GDS domain GDs (SEM), and D GDS It tries to distinguish between real images and fake images.

[0062] For the mapping function G SEM :GDS→SEM and discriminator D SEM , a similar adversarial loss is defined: The generator tries to minimize this loss, while the discriminator tries to maximize it.

[0063] (2) Cycle consistency loss:

[0064] The key to the success of CycleGAN is the supervision using cycle consistency loss, which assumes that the generated images can be transformed back to the original domain. Therefore, for images in the SEM domain, we expect SEM→G GDS (SEM)→G SEM (G GDS (SEM))≈SEM, for images in the GDS domain, it is expected that GDS→G SEM (GDS) → G GDS (G SEM (GDS))≈GDS, so the objective can be expressed as:

[0065]

[0066] Among them, G SEM , G GDS are generators for generating SEM style images and layout style images respectively, ‖.‖1 is the L1 norm, Ε SEM , GDS They represent the summation or integration of all samples sampled from the distribution of the SEM image and GDS image domains, respectively, to calculate the expected value of the function on these samples.

[0067] (3) Identity mapping loss:

[0068] When real samples from the target domain are provided as input, in order to constrain the generator to approach the identity mapping, a pixel-level consistency is introduced between the input image and the generated image. This loss is expressed as:

[0069]

[0070] Among them, G SEM , G GDS are generators for generating SEM style images and layout style images respectively, ‖.‖1 is the L1 norm, Ε SEM , GDSThey represent the summation or integration of all samples sampled from the distribution of the SEM image and GDS image domains, respectively, to calculate the expected value of the function on these samples.

[0071] (4) Position consistency loss:

[0072] In the automatic matching process between the wafer reference layout and the SEM image, the accuracy of image matching is measured by the centroid distribution map of the feature map generated by the wafer feature pattern segmentation branch network Ec. Specifically, the segmented pattern generates the corresponding centroid distribution map. Then, the generated centroid distribution map is compared with the centroid position of the original image, and the L1 norm of the centroid distribution map before and after generation is calculated, and it is expected to be minimized. This goal can be expressed as:

[0073]

[0074] Among them, G SEM , G GDS are generators for generating SEM style images and layout style images, respectively. c (.) is the centroid position distribution diagram after wafer feature pattern segmentation branch network segmentation, ‖.‖1 is the L1 norm, Ε SEM , GDS They represent the summation or integration of all samples sampled from the distribution of the SEM image and GDS image domains, respectively, to calculate the expected value of the function on these samples.

[0075] (5) Edge preservation loss: The wafer pattern line edge detection branch network Ed uses a pre-trained U-Net network to obtain the contours of the input image and the generated image. The edge preservation loss is the L1 distance between the edge of the input image and the generated image. The contour loss is expressed as:

[0076]

[0077] Among them, G SEM , G GDS are the generators for generating SEM style images and layout style images respectively, ‖.‖1 is the L1 norm, E d (.) is the contour information generated by the wafer pattern line edge detection branch network, E SEM , GDS They represent the summation or integration of all samples sampled from the distribution of the SEM image and GDS image domains, respectively, to calculate the expected value of the function on these samples.

[0078] Therefore, the overall objective function can be expressed as:

[0079]

[0080] where λ cyc , im , cen , edge is a parameter that controls the relative importance of each loss.

[0081] S5, training of the transfer model from SEM to layout style images.

[0082] The error of the model is calculated through the loss function, and the model parameters are updated according to the error results, and the model parameters are continuously updated in a loop. The set hyperparameters and loss function weights are evaluated by applying the validation set to the trained model. In this embodiment, the evaluation index is FID, which is based on the difference between the feature representations extracted from the generated image and the real image in the pre-trained Inception network. Finally, the optimal hyperparameter and loss function weight style image is determined.

[0083] S6, Image Matching

[0084] The SEM image to be matched is processed into the input size of the trained SEM to layout style image migration model, and the generator Generator_GDS is used to generate a layout style image; the SIFT algorithm is used to extract features of the generated layout style image and the reference layout to be matched, capture the key feature points in the image, and generate their descriptors. Next, the FLANN matcher is used to match these feature points to find the corresponding relationship between the generated layout style image and the reference layout. The matching process can determine the similarity between the feature points in the generated layout style image and the feature points in the reference layout, and then obtain the geometric transformation relationship between the two, including translation, rotation, scaling, etc. Finally, according to the geometric transformation relationship, the position coordinates of the generated layout style image in the reference layout and the corresponding magnification or reduction multiples are determined, and then the SEM image is processed according to the position coordinates and the magnification or reduction multiples and positioned at the corresponding reference layout position, thereby completing the automatic matching of the SEM image and the reference layout.

[0085] In this embodiment, a system for automatically matching a wafer reference layout and a SEM image is also provided, and the system is used to implement the above-mentioned embodiment. The terms "module", "unit", etc. used below can implement a combination of software and / or hardware for a predetermined function. Although the system described in the following embodiments is preferably implemented in software, it is also possible to implement hardware, or a combination of software and hardware.

[0086] This embodiment provides an automatic matching system for a wafer reference layout and a SEM image, including:

[0087] A data acquisition module, which is used to acquire a SEM image and a reference layout image data set of a wafer pattern, wherein the SEM image and the reference layout image do not need to be paired; and, is used to acquire a SEM image and a reference layout image to be matched;

[0088] A style transfer model module, comprising a cyclic generative adversarial network, a wafer feature pattern segmentation branch network and a wafer pattern line edge detection branch network, wherein the cyclic generative adversarial network comprises two generators and two discriminators, wherein the generator adopts a residual block-based network structure, and the number of residual blocks in the generator for generating SEM style images and the generator for generating layout style images is proportional to the image entropy of the SEM image and the reference layout image;

[0089] The wafer feature pattern segmentation branch network is used to segment the real image and the generated image, label the segmented images in sequence and generate corresponding center of gravity distribution maps; the wafer pattern line edge detection branch network is used to perform wafer pattern line edge detection on the real image and the generated image to generate contour information;

[0090] A style transfer model training module, which is used to train the style transfer model using the SEM images of wafer patterns and the reference layout image dataset. The loss functions used in the training phase include the cyclic generative adversarial network loss, the position consistency loss based on the center of gravity distribution map, and the edge preservation loss based on the contour information.

[0091] The automatic matching module is used to use the trained style transfer model to process the SEM image to be matched into a layout-like style image, match the layout-like style image with the reference layout image to be matched, process the SEM image according to the matching information and locate it to the corresponding reference layout position, and complete the automatic matching process between the SEM image and the reference layout.

[0092] For the system embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment, and the implementation methods of the remaining modules will not be repeated here. The system embodiment described above is only schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of the present invention. Ordinary technicians in this field can understand and implement it without paying creative work.

[0093] The embodiments of the system of the present invention can be applied to any device with data processing capabilities, and the device with data processing capabilities can be a device or apparatus such as a computer. The system embodiments can be implemented by software, or by hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, the corresponding computer program instructions in the non-volatile memory are read into the memory by the processor of any device with data processing capabilities and run.

[0094] Obviously, the above-described embodiments and drawings are only some examples of the present application. For those of ordinary skill in the art, the present application can also be applied to other similar situations based on these drawings without creative work. In addition, it is understandable that, although the work done in this development process may be complicated and lengthy, for those of ordinary skill in the art, certain changes in design, manufacturing or production based on the technical content disclosed in the present application are only conventional technical means and should not be regarded as insufficient content disclosed in the present application. Without departing from the concept of the present application, several variations and improvements can also be made, which all belong to the scope of protection of the present application. Therefore, the scope of protection of the present application shall be subject to the attached claims.

Claims

1. A method for automatically matching a wafer reference layout and a SEM image, characterized in that: The following steps are involved: Step 1, obtaining a SEM image of a wafer pattern and a reference layout image dataset, wherein the SEM image and the reference layout image do not need to be paired; Step 2, constructing a style transfer model from SEM to layout style image, including a cyclic generative adversarial network, a wafer feature pattern segmentation branch network and a wafer pattern line edge detection branch network, wherein the cyclic generative adversarial network includes two generators and two discriminators, and the generator adopts a network structure based on residual blocks, and the number of residual blocks in the generator for generating SEM style image and the generator for generating layout style image is proportional to the image entropy of SEM image and reference layout image; The wafer feature pattern segmentation branch network is used to segment the real image and the generated image, label the segmented images in sequence and generate corresponding center of gravity distribution maps; the wafer pattern line edge detection branch network is used to perform wafer pattern line edge detection on the real image and the generated image to generate contour information; Step 3, using the image dataset obtained in step 1 to train the style transfer model, the loss functions used in the training phase include cyclic generative adversarial network loss, position consistency loss based on centroid distribution map, and edge preservation loss based on contour information; The formulas for the position consistency loss based on the centroid distribution map and the edge preservation loss based on the contour information are as follows: in, are position consistency loss and edge preservation loss respectively, SEM and GDS are the real SEM image and reference layout image respectively, G SEM , G GDS are generators for generating SEM style images and layout style images, respectively. c (.) is the centroid position distribution diagram of the wafer feature pattern segmentation branch network after segmentation, ‖.‖1 is the L1 norm, E d (.) is the contour information generated by the wafer pattern line edge detection branch network, E sEM , GDS Respectively, they represent the summation or integration of all samples sampled from the distribution of the SEM image and GDS image domains, which are used to obtain the expected value; Step 4: Use the trained style transfer model to process the SEM image to be matched into a layout-like style image, match the layout-like style image with the reference layout image to be matched, process the SEM image according to the matching information and locate it to the corresponding reference layout position, and complete the automatic matching process between the SEM image and the reference layout.

2. The automatic matching method of wafer reference layout and SEM image according to claim 1, characterized in that: The generator includes an initialized convolution layer, a downsampling layer, several residual blocks, an upsampling layer and an output layer connected in sequence, and the upsampling layer and the downsampling layer both include several convolution layers and an attention module located after the last convolution layer.

3. The automatic matching method of wafer reference layout and SEM image according to claim 1, characterized in that: The discriminator includes several convolution blocks, several attention modules, and an output layer. The attention module is located between two adjacent convolution blocks, and the convolution block is composed of several convolution layers.

4. The automatic matching method of wafer reference layout and SEM image according to claim 1, characterized in that: Before the SEM image is input into the style transfer model, the method further includes a step of preprocessing the SEM image, wherein the preprocessing includes denoising and high-resolution processing.

5. The automatic matching method of wafer reference layout and SEM image according to claim 1, characterized in that: The wafer feature pattern segmentation branch network adopts a pre-trained TernausNet network.

6. The automatic matching method of wafer reference layout and SEM image according to claim 1, characterized in that: The wafer pattern line edge detection branch network adopts a pre-trained U-Net network, and the labels used in the pre-training process are contour information extracted after wafer pattern line edge detection is performed on SEM images and reference layout images through the Canny algorithm.

7. The automatic matching method of wafer reference layout and SEM image according to claim 1, characterized in that: The cycle generative adversarial network loss includes adversarial loss, cycle consistency loss and identity mapping loss.

8. The automatic matching method of wafer reference layout and SEM image according to claim 1, characterized in that: The step 4 is specifically as follows: The SEM image to be matched is processed into the input size of the trained style transfer model, and a layout-like style image is generated by using a generator for generating layout-style images; the SIFT algorithm is used to extract key feature points of the generated layout-like style image and the reference layout to be matched, and the key features are matched using a FLANN matcher to obtain the geometric transformation relationship between the layout-like style image and the reference layout; based on the geometric transformation relationship, the position coordinates and scaling factor of the layout-like style image in the reference layout are obtained; finally, the SEM image is processed according to the position coordinates and scaling factor and positioned at the corresponding reference layout position, thereby completing the automatic matching process between the SEM image and the reference layout.

9. An automatic matching system for wafer reference layout and SEM image, characterized in that: include: A data acquisition module, which is used to acquire a SEM image of a wafer pattern and a reference layout image data set, wherein the SEM image and the reference layout image do not need to be paired; and, for obtaining a SEM image and a reference layout image to be matched; A style transfer model module, comprising a cyclic generative adversarial network, a wafer feature pattern segmentation branch network and a wafer pattern line edge detection branch network, wherein the cyclic generative adversarial network comprises two generators and two discriminators, wherein the generator adopts a residual block-based network structure, and the number of residual blocks in the generator for generating SEM style images and the generator for generating layout style images is proportional to the image entropy of the SEM image and the reference layout image; The wafer feature pattern segmentation branch network is used to segment the real image and the generated image, label the segmented images in sequence and generate corresponding center of gravity distribution maps; the wafer pattern line edge detection branch network is used to perform wafer pattern line edge detection on the real image and the generated image to generate contour information; A style transfer model training module, which is used to train the style transfer model using the SEM images of wafer patterns and the reference layout image dataset. The loss functions used in the training phase include the cyclic generative adversarial network loss, the position consistency loss based on the center of gravity distribution map, and the edge preservation loss based on the contour information. The automatic matching module is used to use the trained style transfer model to process the SEM image to be matched into a layout-like style image, match the layout-like style image with the reference layout image to be matched, process the SEM image according to the matching information and locate it to the corresponding reference layout position, and complete the automatic matching process between the SEM image and the reference layout.

Images