Sample observation device and method

Abstract

When the ideal image quality differs depending on the region of the estimated image, it is difficult to learn to estimate images of different image quality for each region using a single loss function. A method and apparatus for observing a sample in which a first learning image and a second learning image corresponding to the first learning image are acquired, and an estimation processing parameter of an estimation engine for estimating the second learning image from the first learning image is learned using the first learning image and the second learning image. In the learning of the estimation processing parameters, an estimation image estimated from a first learning image and a second learning image corresponding to the first learning image are divided into regions Ri. The learning is performed using a loss function Fi that evaluates the loss of the pixel group Pi of the second learning image and the pixel group Qi of the estimated image, which are included in each region Ri, with a predetermined reference, where i = 1-N, where N is the number of regions.

Description

Technical Field

[0001] The present invention relates to a method and apparatus for photographing a sample such as a semiconductor wafer using a charged particle microscope or the like to obtain an image for sample observation, and to a method and apparatus for estimating a higher quality image from the photographed image. The apparatus and method are provided with a method for estimating an image whose image quality varies depending on the region when the correspondence between a first learning image and a second learning image is learned in advance by a machine learning method. Background Art

[0002] In the manufacture of semiconductor wafers, it is important to start the manufacturing process quickly and shift to a high-yield mass production system in advance in order to ensure profitability. For this purpose, various inspection devices, observation devices, and measurement devices are introduced into the production line.

[0003] The sample observation device is a device that captures the defect position on the wafer with high resolution and outputs an image based on the defect position coordinates (coordinate information indicating the position of the defect on the sample) output by the inspection device. The sample observation device (hereinafter referred to as the review SEM) using a scanning electron microscope (SEM: Scanning Electron Microscope) is widely used. In the mass production line of semiconductors, it is desired to automate the observation work. The review SEM is equipped with a function of automatic defect image collection and processing (ADR: Automatic Defect Review) that automatically collects images of the defect position in the sample.

[0004] There are many types of structures of circuit patterns formed on semiconductor wafers, and the defects generated also have various types, generation locations, etc. It is important to capture and output high-visibility high-quality images of defects, circuit patterns, etc. Therefore, for the original captured image obtained by imaging the signal obtained from the detector of the review SEM, the image for sample observation is obtained by improving the visibility using image processing technology. As a method of improving visibility, a plurality of methods of pre-learning the correspondence between images of different image qualities and inferring the image of the image quality of the other party when an image of the same image quality as that of one party is input are proposed. For example, the following method is described in Japanese Patent Gazette No. 2018-137275 (Patent Document 1): by pre-learning the correlation between an image captured at a low magnification and an image captured at a high magnification, a high magnification image is inferred from a low magnification image.

[0005] Prior art literature

[0006] Patent Literature

[0007] Patent Document 1: Japanese Patent Application Publication No. 2018-137275

[0008] Non-patent literature

[0009] Non-patent literature 1: Dong, Chao, et al. "Image super-resolution using deep convolutional networks." arXiv preprint arXiv:1501.00092 (2014). Summary of the invention

[0010] Problems to be solved by the invention

[0011] In a sample observation device, in order to observe defects or circuit patterns, etc., it is important to output images with high visibility of them, and image processing is applied to the captured image to improve the visibility of the image. One method is to pre-learn the relationship between the captured image and the high-quality image by a machine learning method, so as to infer the high-quality image with the captured image as input. Usually, in the machine learning method, the loss of the estimated image and the high-quality image is calculated by a pre-defined single loss function, and the estimated processing parameters are updated to reduce the loss. However, when observing, the area to be observed varies depending on the observer, so sometimes the image quality required for each area is different. In this case, it is difficult to learn by using a single loss function to infer images of different qualities for each area.

[0012] Means for solving problems

[0013] In order to solve the above-mentioned problems, the present invention provides a visual inspection method and a visual inspection system having the following features.

[0014] That is, in the method and apparatus for observing the sample,

[0015] acquiring a first learning image and a second learning image corresponding to the first learning image,

[0016] using the first learning image and the second learning image, learning an estimation processing parameter of an estimation engine for estimating the second learning image from the first learning image,

[0017] In learning the estimation processing parameters, an estimated image estimated from a first learning image and a second learning image corresponding to the first learning image are divided into regions Ri (i=1 to N, N: number of regions).

[0018] The learning is performed using a loss function Fi that evaluates the loss of the pixel group Pi of the second learning image and the pixel group Qi of the estimated image included in each region Ri based on a predetermined criterion.

[0019] Effects of the Invention

[0020] According to the present invention, the method of calculating the loss in learning of the estimation engine can be changed for each region, thereby improving the performance of the image estimation engine. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 This is a diagram showing an embodiment of a learning process sequence of the estimation engine of the present invention.

[0022] Figure 2 It is used to illustrate Figure 1 The figure shows the details of the area segmentation processing of this embodiment.

[0023] Figure 3 This is a diagram illustrating an image obtained by photographing a three-layer circuit pattern in Example 1.

[0024] Figure 4 This is a diagram for illustrating Example 1, in which an estimated image corresponding to a first learning image and a second learning image are divided into a defective region R1” and a normal region R2” without defects, and loss function F1” is used in R1” and loss function F2” is used in R2” to learn estimated processing parameters.

[0025] Figure 5 This is a diagram showing a configuration of acquiring a label image by assigning labels to each pattern and background portion using layout information acquired from design data in the first embodiment.

[0026] Figure 6 This is a diagram for explaining the loss function Fi of Example 1. The loss function Fi is defined by the weighted sum of element losses calculated by multiple element loss functions fij (j=1 to M, M is the number of element losses), and the weight wij of the element loss value is changed for each region Ri of the second learning image.

[0027] Figure 7 This is a diagram showing an example of a graphical user interface (GUI) according to this embodiment.

[0028] Figure 8 This is a diagram showing an example of a neural network having a three-layer structure according to this embodiment.

[0029] Fig. 9 1 and 2 are diagrams showing a first learning image captured at an added frame number of 1 and a second learning image captured at an added frame number of 64 according to the present embodiment. DETAILED DESCRIPTION

[0030] Hereinafter, embodiments for implementing the present invention will be described with reference to the accompanying drawings. In addition, the embodiments described below do not limit the invention involved in the scope of the claimed patent protection, and the elements and their combinations described in the embodiments are not necessarily all necessary for the solution means of the invention.

[0031] Example 1

[0032] The present embodiment is an embodiment of a method and apparatus for observing a sample, wherein a first learning image and a second learning image corresponding to the first learning image are obtained, and the first learning image and the second learning image are used to learn estimation processing parameters of an estimation engine that estimates the second learning image based on the first learning image. In the learning of the estimation processing parameters, the estimated image estimated based on the first learning image and the second learning image corresponding to the first learning image are divided into regions Ri (i=1 to N, N: number of regions), and learning is performed using a loss function Fi that evaluates the losses of the pixel groups Pi of the second learning image and the pixel groups Qi of the estimated image contained in each region Ri based on a predetermined benchmark. (1)

[0034] That is, Example 1 is an example of the following structure:

[0035] acquiring a first learning image and a second learning image corresponding to the first learning image,

[0036] using the first learning image and the second learning image, learning an estimation processing parameter of an estimation engine for estimating the second learning image from the first learning image,

[0037] In learning the estimation processing parameters, an estimated image estimated from a first learning image and a second learning image corresponding to the first learning image are divided into regions Ri (i=1 to N, N: number of regions).

[0038] The learning is performed using a loss function Fi that evaluates the loss between the pixel group Pi of the second learning image and the pixel group Qi of the estimated image included in each region Ri based on a predetermined criterion.

[0039] This feature is supplemented. The estimation engine updates the estimation parameters based on the loss, so the image quality of the image output by the estimation engine after learning changes according to the loss function used during learning. In the previous method, a single loss function is applied to the entire image to learn the parameters of the estimation engine, so it is difficult to learn in a way that images of different image qualities are estimated according to the region. In this embodiment, by changing the loss function for each region, images of different image qualities can be estimated for each region.

[0040] The features of this embodiment are described in more detail below.

[0041] Figure 1 An example of a learning process sequence of the estimation engine of this embodiment is shown. In this figure, a sample is photographed and a plurality of pairs of first learning images (100) and second learning images (101) are acquired.

[0042] Here, when obtaining the first learning image and the second learning image, one or more of the image resolution, the number of added frames, and the focus position are changed to obtain the second learning image so that its quality is higher than that of the first learning image. Fig. 9 As shown, a first learning image (900) captured with an added frame number of 1 and a second learning image (901) captured with an added frame number of 64 may be used.

[0043] Next, the first learning image and the second learning image are aligned to obtain the aligned first learning image (103) and the aligned second learning image (106). In addition, in the alignment, normalized correlation or pixel difference is used as an evaluation value, and the alignment is performed according to the position where the evaluation value is maximum or minimum. In addition, when the image resolutions of the first learning image and the second learning image are different, the image resolutions can be made consistent by linear interpolation before alignment, and then the alignment is performed.

[0044] The first learning image (103) after the alignment is input to the estimation engine (104) to obtain the estimated image (105). The region segmentation process (107) is applied to the second learning image (106) after the alignment to obtain the region R1 (108) to the region RN (109). Next, for each region Ri (i=1 to N, N: the number of regions), a different loss function Fi is used to calculate the loss between the pixel group Pi of the second learning image 106 after the alignment and the pixel group Qi of the estimated image 105.

[0045] That is, in region R1, the loss is calculated by loss function F1, and in region RN, the loss is calculated by loss function FN different from F1. The loss (110, 111) of each region is added together to calculate the loss (112) of the entire image, and the estimation processing parameters of the estimation engine (104) are updated based on the loss.

[0046] In addition, various existing machine learning methods can be used as the estimation engine (104), for example, deep neural networks, etc.

[0047] As a method of using a deep neural network, the convolutional neural network described in Non-Patent Document 1 can be used. Specifically, Figure 8A neural network with a three-layer structure as shown in the figure can be used. Here, Y represents the input image, F1(Y) and F2(Y) represent intermediate data, and F(Y) is the estimated result. In addition, the intermediate data and the final result are calculated by equations 1 to 3. Among them, "*" represents a convolution operation. Here, W1 is n1 filters of size c0×f1×f1, c0 represents the number of channels of the input image, and f1 represents the size of the spatial filter. By convolving the input image with a filter of size n0×f1×f1 n1 times, an n1-dimensional feature map is obtained. B1 is an n1-dimensional vector, which is the bias component corresponding to the n1 filters. Similarly, W2 is a filter of size n1×f2×f2, B2 is an n2-dimensional vector, W3 is a filter of size n2×f3×f3, and B3 is a c3-dimensional vector.

[0048] F1(Y)=max(O, W1*Y+B1)…(Formula 1)

[0049] F2(Y)=ma×(O, W2*F1(Y)+B2)…(Formula 2)

[0050] F(Y)=W3*F2(Y)+B3…(Formula 3)

[0051] Among them, c0 and c3 are values ​​determined by the number of channels of the first learning image and the second learning image. In addition, f1, f2, n1, and n2 are hyperparameters determined by the user before the learning sequence, for example, f1 = 9, f2 = 5, n1 = 128, and n2 = 64. The parameters adjusted by the learning of the inference engine (104) are W1, W2, W3, B1, B2, and B3.

[0052] There are various methods for region segmentation depending on the purpose. Specific examples of region segmentation methods include the following.

[0053] (A1) Segmentation of edge and non-edge regions

[0054] (A2) Division of the upper pattern area and the lower pattern area

[0055] (A3) Segmentation of defective area and normal (non-defective) area

[0056] (A4) Label the design data in advance and segment the area based on the labels

[0057] In this specification, (A1) to (A4) described above are described as examples of region division, but region division is not limited thereto, and region division can be performed by any method based on the application or user designation. (2)

[0059] In images for sample observation, it is important to estimate images with high contrast or fine features in edge areas such as the outline of a circuit pattern. On the other hand, in order to avoid misidentifying flat portions without bumps (non-edge areas) as defects or circuit patterns, it is important to estimate images with low contrast or from which fine elements such as noise have been removed.

[0060] In response to this problem, in addition to the above-mentioned feature (1), the present embodiment is characterized in that, in the region segmentation processing (104), a brightness gradient image is obtained by applying a differential filter to the second learning image, and the estimated image corresponding to the first learning image and the second learning image are segmented into an edge region R1 and a non-edge region R2 (A1) of the circuit pattern using the brightness gradient image and the edge judgment threshold, and the loss function F1 is used in R1 and the loss function F2 is used in R2 to learn the estimated processing parameters.

[0061] use Figure 2 Supplement this feature.

[0062] In the figure, a brightness gradient image (203) is obtained by applying a differential filter (202) to the second learning image (201). The differential filter can use various methods. For example, the absolute value of the difference between the object pixel and the adjacent pixel on the right and the absolute value of the difference between the object pixel and the adjacent pixel below can be used as the differential filtering result. Next, in the edge judgment process (205), the brightness gradient image (203) and the edge judgment threshold (204) are used to binarize the brightness gradient image (203), thereby dividing it into an edge area R1 (206) and a non-edge area R2 (207). In addition, the edge judgment threshold (204) can also be set by the user or according to a predetermined rule. Using the above-mentioned features, different loss functions can be used for learning for edge areas and non-edge areas, and learning can be performed to infer images with different image quality in edge areas and non-edge areas. (3)

[0064] When the sample has a circuit pattern with multiple layers in the sample observation image, it is sometimes necessary to estimate an image with high contrast in a specific layer. To address this issue, in this embodiment, in addition to the features described in (1) and (2), in the region segmentation process (107), the estimated image corresponding to the first learning image and the second learning image are segmented into a region R1' of the first layer pattern to a region RN' of the Nth layer pattern using a second learning image and a layer determination threshold, and the estimated processing parameters are learned using a loss function Fi' in Ri' (i=1 to N, N: number of regions).

[0065] Regarding this feature (3), Figure 3The following supplementary explanation is given by taking an image obtained by photographing a three-layer circuit pattern as an example.

[0066] The second learning image (301) and the layer determination threshold (302) are input to the layer determination process (303), whereby the layer is divided into a 1-layer pattern region R1' (304), a 2-layer pattern region R2' (305), and a 3-layer pattern region R3' (305).

[0067] For example, when a semiconductor wafer having multiple layers of circuit patterns is photographed by a charged particle microscope, an image is generally obtained in which the upper layers have brighter patterns and the lower layers have darker patterns. Therefore, in the layer determination process, the brightness of the second learning image (301) can be N-valued based on the layer determination threshold (302) for separation. According to this feature, by using a different loss function for each area divided by layer for learning, it is possible to learn in a manner that estimates images with different image quality depending on the layer, that is, images with high contrast in a specific layer. (4)

[0069] In the sample observation image, when there are defects in the sample, the visibility of the defect area is important, and sometimes an image with high contrast of the defect is required. In addition, in order to avoid misidentification of defects in areas without defects, it is important to estimate images with less noise, and sometimes the items that are emphasized in image estimation vary depending on the presence or absence of defects.

[0070] To address this problem, in this embodiment, in addition to the above-mentioned features, it is also characterized in that:

[0071] Obtain a first learning image and a second learning image based on the defect coordinate information,

[0072] Obtaining a reference image that corresponds to the second learning image and does not contain defects,

[0073] The defect area in the second learning image is detected using the second learning image and the reference image. According to the detected area, the estimated image corresponding to the first learning image and the second learning image are divided into a defect area R1” and a normal area R2” without defects. In R1”, the loss function F1” is used to learn the estimated processing parameters, and in R2”, the loss function F2” is used to learn the estimated processing parameters.

[0074] use Figure 4 This supplements feature (4).

[0075] A second learning image (401) and a reference image (402) obtained by photographing an area where a circuit pattern identical to that of the second learning image is formed are obtained, defective areas are detected (403) by comparing the second learning image with the reference image, and the second learning image is divided into defective areas R1" (404) and non-defective areas R2" (405).

[0076] According to this feature, learning using different loss functions can be performed in the defect region R1 ″ and the defect-free region R2 ″, and learning can be performed to estimate an image with high visibility of defects and little noise in the defect-free region. (5)

[0078] In the sample observation image, it is sometimes necessary to estimate the image with high contrast in a specific pattern. In order to solve this problem, in addition to the above-mentioned features, the present invention also has the following features: assigning labels to the design data, aligning the position of one of the first learning image and the second learning image with the design data, dividing the second learning image into regions Ri (i=1 to N, N: number of regions) based on the alignment result and the label, and using the loss function Fi'' in the region Ri'' to learn and estimate the processing parameters.

[0079] Here, the design data refers to layout data on the observed sample. For example, if the sample is a semiconductor, the design data is data in which edge information of a design shape of a semiconductor circuit pattern is written as coordinate data.

[0080] use Figure 5 This supplements feature (5).

[0081] Using the layout information (501) obtained from the design data, labels are assigned to each pattern or background portion (502), and label images (503) are obtained. Figure 5 In the example, three labels are assigned. In addition, the design data (501) and the second learning image (504) are compared to obtain the position (506)(505) corresponding to the second learning image. Based on the design data (506) and the label image (503) corresponding to the second learning image, the second learning image (504) is subjected to region segmentation processing (507), thereby obtaining regions R1'" to R3'" (508 to 510).

[0082] In the corresponding position acquisition process, for example, the contour information of the pattern may be acquired by performing edge extraction from the second learning image, and the corresponding position may be searched by comparing with the design data.

[0083] According to this feature, by performing learning using a different loss function for each region corresponding to a label given to the design data, it is possible to perform learning so as to estimate an image with a high contrast of a specific pattern. (6)

[0085] Regarding image quality, it is difficult to express it with a single loss function when there are various factors such as brightness and contrast. Therefore, in this embodiment, in addition to the feature described in (1), it is characterized in that the loss function Fi is defined by the weighted sum of the element losses calculated by multiple element loss functions fij (j=1 to M, M is the number of element losses), and the weight wij of the element loss value is changed for each region Ri of the second learning image.

[0086] use Figure 6 This supplements the feature (6). In addition, although the loss F1 (P1, Q1) is used as an example for explanation, it can be applied to any loss Fi (Pi, Qi).

[0087] The estimated image (600) and the second learning image are used as inputs to calculate the element loss functions F11 to F1M (602 to 603). The weights w11 to w1M (604) of each element loss indicating which element loss is emphasized and each element loss (602 to 603) are used as inputs to calculate the loss F1 (P1, Q1) (605).

[0088] Here, in the calculation of the loss F1 (P1, Q1), it is calculated by the weighted sum of the losses of the respective elements shown in the following equation.

[0089] F1(P1, Q1)=w11f11(P1, Q1)+w12f12(P1Q1)+…+w1Mf1M(P1, Q1)

[0090] The element loss function refers to a method of calculating the loss (element loss) corresponding to each element of image quality. For example, in the loss function Fi, the element loss function related to brightness is represented by the absolute square error (Pi-Qi)^2, and the element loss function related to contrast is represented by the absolute square error of the brightness gradient (Pi'-Qi')^2. In addition, Pi' is the brightness gradient of the pixel group Pi, and Qi' is the brightness gradient of the pixel group Qi.

[0091] According to the above characteristics, by calculating the weighted sum of the multiple element losses calculated by the multiple element loss functions for each region Ri and learning, it is possible to perform learning in a manner that estimates an image that reflects multiple elements related to image quality. (7)

[0093] Figure 7 An example of a graphical user interface (GUI) when implementing this embodiment is shown.

[0094] In this GUI, it is possible to designate the setting of the determination threshold in area segmentation and the importance of each area.

[0095] The second learning image (700) is segmented into regions to display a defect region (701), an edge region (702), a non-edge region (703), a first layer region (704), and a second layer region (705). In addition, the thresholds (706, 707) used in the region segmentation can be set. In addition, the settings (708, 710, 712) of each region and the importance (709, 711, 713) of each element in each region can be set. By setting the weight (604) of the loss function of each element based on the importance set here, it can be reflected in the loss.

[0096] In addition, the present invention is not limited to the above-described embodiments, and includes various modifications. For example, the above-described embodiments and modifications are described in detail to easily explain the present invention, and are not limited to necessarily having all the described configurations.

Claims

1. A method and device for observing a sample, characterized in that: acquiring a first learning image and a second learning image corresponding to the first learning image, using the first learning image and the second learning image, learning an estimation processing parameter of an estimation engine for estimating the second learning image from the first learning image, In learning the estimation processing parameters, an estimated image estimated from a first learning image and a second learning image corresponding to the first learning image are divided into regions Ri, where i=1 to N, N being the number of regions. The learning is performed using a loss function Fi that evaluates the loss of the pixel group Pi of the second learning image and the pixel group Qi of the estimated image included in each region Ri based on a predetermined criterion.

2. The method and device for observing a sample according to claim 1, characterized in that: When capturing the first learning image and the second learning image, When one or more of the conditions of image resolution, number of added frames, and focus position are changed, the quality of the second learning image is higher than that of the first learning image.

3. The method and device for observing a sample according to claim 2, characterized in that: Using convolutional neural networks in the inference engine, The estimation processing parameters are updated by the error back-propagation process so that the loss calculated by the loss function becomes smaller.

4. The method and device for observing a sample according to claim 3, characterized in that: A brightness gradient image is obtained by applying a differential filter to the second learning image. The brightness gradient image and the edge judgment threshold are used to divide the estimated image based on the first learning image and the second learning image into an edge area R1 and a non-edge area R2 of the circuit pattern. The loss function F1 is used to learn the estimated processing parameters in R1, and the loss function F2 is used to learn the estimated processing parameters in R2.

5. The method and device for observing a sample according to claim 3, characterized in that: Using the second learning image and the layer determination threshold, the estimated image estimated based on the first learning image and the second learning image are divided into area R1' of the first layer pattern to area RN' of the Nth layer pattern, and the loss function Fi' is used in Ri' to learn the estimated processing parameters, where i=1~N, and N is the number of areas.

6. The method and device for observing a sample according to claim 3, characterized in that: Obtain a first learning image and a second learning image based on the defect coordinate information, Obtaining a reference image that corresponds to the second learning image and does not contain defects, The defect area in the second learning image is detected using the second learning image and the reference image. According to the detected area, the estimated image estimated based on the first learning image and the second learning image are divided into a defect area R1" and a normal area R2" without defects. The loss function F1"' is used in R1" to learn the estimated processing parameters, and the loss function F2"' is used in R2" to learn the estimated processing parameters.

7. The method and device for observing a sample according to claim 3, characterized in that: Assign labels to design data. aligning one of the first learning image and the second learning image with the design data, The second learning image is divided into regions Ri'' based on the position alignment result and the label, wherein i=1-N, N is the number of regions, The loss function Fi'' is used in the region Ri'' to learn the estimated processing parameters.

8. The method and device for observing a sample according to any one of claims 3 to 7, characterized in that: The loss function Fi is defined as a weighted sum of element losses calculated by a plurality of element loss functions fij, with different weights for each region Ri of the second learning image, where j=1 to M, and M is the number of element losses.

Images