Image segmentation method and image segmentation system

Abstract

In order to realize segmentation technology for a captured image to reduce a burden on a worker, the present disclosure executes acquiring a learning image from a learning image group when learning a processing parameter for a segmentation engine, specifying a first region in the learning image, acquiring a first partial image of the learning image corresponding to the first region, searching a similar learning image including a second partial image similar to the first partial image from images included in the learning image group and acquiring a second region indicating the second partial image, acquiring a label map and a confidence map corresponding to each of the first region and the second region by using the segmentation engine partway through learning, the learning image, and the similar learning image, and updating the processing parameter of the segmentation engine such that the label map corresponding to the first region is closer to the label map corresponding to the second region (see Fig. 2).

Description

Image segmentation method and image segmentation system

[0001] The present disclosure relates to an image segmentation method and system.

[0002] In semiconductor wafer manufacturing, it is important to quickly set up the manufacturing process and quickly transition to a high-yield mass production system in order to ensure profits. For this purpose, various inspection, observation, and measurement devices are introduced into the manufacturing line.

[0003] A sample observation device is a device that captures high-resolution images of defect positions on a wafer based on defect position coordinates (coordinate information indicating the positions of defects on the sample) output by an inspection device, and outputs the images. Sample observation devices using a scanning electron microscope (SEM) (hereinafter referred to as review SEM) are widely used. Automating observation work is desirable in semiconductor mass production lines, and review SEMs are equipped with a function for performing automatic defect image review (ADR), which automatically collects images of defect positions within a sample. There are various methods for detecting the positions of defects. For example, Patent Document 1 discloses a method for identifying defect positions by estimating an image that does not include a defect from an image that includes the defect.

[0004] Furthermore, in order to observe defects that occur during semiconductor manufacturing, it is important to capture and output high-quality images with high visibility of defects, circuit patterns, etc. On the other hand, if structures such as the position and shape of defects, circuit patterns, etc. can be grasped in raw captured images obtained by visualizing signals obtained from the detector of a review SEM, images with higher visibility of defects and circuit patterns can be obtained, for example, by applying image processing appropriate for each of these structures. As one method for grasping structures such as the position and shape of defects, circuit patterns, etc., many methods have been proposed, in which pairs of what is captured in each pixel of a captured image (ground truth data) are prepared in advance, the relationship between the image and the ground truth data is learned, and then what is captured in each pixel when the captured image is input is estimated. For example, Non-Patent Document 1 discloses a method for estimating what is captured in each pixel, such as a building, tree, or road, using an input image.

[0005] Patent Publication No. 2021-141231

[0006] Ronneberger et al. "U-Net: Convolutional Networks for Biomedical Image Segmentation" Medical Image Computing and Computer-Assisted Intervention (MICCAI), Springer, LNCS, Vol.9351: 234--241, 2015

[0007] It is important for sample observation devices to output images with high visibility in order to observe defects, circuit patterns, etc., and image segmentation is an effective way to improve visibility. One method for doing this is to input the captured image and segment it, using machine learning techniques to learn in advance the relationship between the captured image and ground truth data that indicates what is depicted in each pixel of the captured image.

[0008] However, to obtain correct data corresponding to a captured image, an operator must provide known correct data, which increases the operator's workload and can make the process difficult. In view of this situation, the present disclosure proposes a captured image segmentation technique that reduces the operator's workload.

[0009] In order to solve the above-mentioned problems, the present disclosure provides, as an example, an image segmentation method for assigning a desired label to each pixel of an input image by a computer system, the method including: acquiring a group of training images including one or more images; using the group of training images to learn processing parameters of a segmentation engine that outputs a label map in which each pixel is labeled and a confidence map indicating the probability that each pixel belongs to each label; performing a segmentation process on the input image using the segmentation engine obtained by learning the processing parameters to obtain the label map of the input image, wherein learning the processing parameters of the segmentation engine includes: acquiring a training image from the group of training images and specifying a first region in the training image; acquiring a first partial image of the training image that corresponds to the first region; searching for a similar training image including a second partial image similar to the first partial image from images included in the group of training images, and acquiring a second region indicating the second partial image; The present invention proposes an image segmentation method, including: obtaining a label map and a confidence map corresponding to the first region and the second region, respectively, using the segmentation engine in the middle of training, the training images, and the similar training images; and updating the processing parameters of the segmentation engine so that the label map corresponding to the first region approaches the label map corresponding to the second region.

[0010] Further features related to the present disclosure will become apparent from the description of this specification and the accompanying drawings. Also, aspects of the present disclosure are achieved and realized by the elements and combinations of various elements and the aspects of the following detailed description and the appended claims. The description of this specification is merely exemplary and does not limit the scope or application of the claims of the present disclosure in any way.

[0011] According to the technology of the present disclosure, the segmentation engine can be trained using only captured images, which reduces the burden on the worker.

[0012] FIG. 1 is a diagram illustrating an example of a schematic configuration of a segmentation system 1 (which can be used in each example) according to an embodiment of the present disclosure. FIG. 2 is a flowchart illustrating a sequence of a process for training a segmentation engine and a process for applying the trained segmentation engine during sample observation. FIG. 3 is a flowchart illustrating details of the training process (step S103) of the segmentation engine 1011. FIG. 4 is a flowchart illustrating an improvement plan 1 of the training process (step S103) according to Example 2. FIG. 5 is a flowchart illustrating an improvement plan of the segmentation process (details of step S105) according to Example 3. FIG. 6 is a diagram illustrating an example of the configuration of a neural network for estimating a label map in Example 3. FIG. 7 is a flowchart illustrating an improvement plan 2 of the training process (step S103) according to Example 4. FIG. 8 is a flowchart illustrating details of the similarity search process according to Example 5. FIG. 9 is a diagram illustrating label map generation based on the presence or absence of defects according to Example 6. FIG. 10 is a flowchart illustrating an improvement plan 3 of the training process (step S103) of the segmentation engine 1011 according to Example 7. 14 is a flowchart for explaining an improvement plan 4 of the learning process (step S103) of the segmentation engine 1011 according to Example 8. FIG. 15 is a diagram showing an example of a change in a captured image. FIG. 16 is a flowchart for explaining an improvement plan 5 of the learning process (step S103) of the segmentation engine 1011 according to Example 9. FIG. 17 is a flowchart for explaining an improvement plan 6 of the learning process (step S103) of the segmentation engine 1011 according to Example 10. FIG. 18 is a diagram showing an example of the configuration of a GUI 1400 for correcting at least one of the position of region R2 or the label map according to Example 11.

[0013] This embodiment relates to a method and apparatus for segmenting captured images obtained by imaging a sample such as a semiconductor wafer using a charged particle microscope (e.g., SEM), and proposes training the segmentation engine and performing segmentation processing using only the captured images, without the need for an operator to provide correct data.

[0014] Hereinafter, embodiments and examples of the present disclosure will be described with reference to the accompanying drawings. In the accompanying drawings, functionally identical elements may be designated by the same numerals. Note that the accompanying drawings show specific embodiments and implementation examples in accordance with the principles of the present disclosure, but these are intended to aid in understanding the present disclosure and are by no means to be used to interpret the present disclosure in a limiting manner.

[0015] In a first embodiment, a computer system for dividing an input image into desired regions includes acquiring a group of training images including one or more training images, using the group of training images to learn processing parameters of a segmentation engine that outputs a label map labeled for each pixel and a confidence map indicating the probability that each pixel belongs to each label, and inputting an image into the trained segmentation engine to acquire the label map. Learning the segmentation parameters includes specifying a region R1 of the training image, searching the group of training images for similar training images that include a second partial image similar to a first partial image in the training image indicated by region R1, and calculating a region R2 indicating the second partial image (similar region search), using the segmentation engine, the training image, and the similar training image in the middle of training to acquire a label map and a confidence map corresponding to regions R1 and R2, respectively, and updating the processing parameters of the segmentation engine so that the label map corresponding to region R1 approaches the label map corresponding to region R2 (comparing R1 and R2). The processing parameter update is performed when the confidence level of the label map corresponding to region R1 is lower than the confidence level of the label map corresponding to region R2.

[0016] <Schematic Configuration Example of Segmentation System> FIG. 1A is a diagram showing a schematic configuration example of a segmentation system 1 (which can be used in each example) according to an embodiment of the present disclosure.

[0017] The segmentation system 1 includes a segmentation device 10 that segments a target image, and a charged particle beam device 20 such as a scanning electron microscope (SEM) that captures the target image for segmentation.

[0018] The segmentation device 10 includes a processor 101 configured by, for example, a CPU or an MPU, a communication device 102, a storage device 103, an input device 104, and an output device 105.

[0019] The communication device 102 acquires an image to be segmented from the charged particle beam device 20 or a server (not shown) and stores it in the storage device 103. The storage device 103 stores not only the acquired image but also various programs (including a segmentation program), various parameters, data, and the like. The input device 104 is configured, for example, by a keyboard or a mouse. The input device 104 allows an operator (user) to input predetermined instructions and directly input (modify) parameters. The output device 105 is configured, for example, by a display device or a printer, and can display the image to be segmented and the segmentation results.

[0020] The processor 101 reads programs necessary for segmentation processing and learning from the storage device 103, expands the programs in an internal memory (not shown), and generates various processing units. Specifically, the various processing units include a segmentation engine 1011, a region R1 determination processing unit 1012, a cutout processing unit 1013, a similar region search processing unit 1014, a learning necessity determination unit 1015, a segmentation engine update processing unit 1016, a high-confidence region search processing unit 1017, a success / failure determination processing unit 1018, a dissimilar region search processing unit 1019, a defect detection processing unit 1020, and a defect presence / absence determination processing unit 1021. The operation of the various processing units will become clear from the description of each embodiment.

[0021] 1B is a flowchart for explaining a sequence of processing for training the segmentation engine 1011 and processing for applying the trained segmentation engine 1011 during sample observation. Note that the sequence of FIG. 1B will be explained here with the processor 101 acting as the subject of operations in each step.

[0022] (i) Step S100 The processor 101 acquires a plurality of sample images captured by the charged particle beam device 20 as a group of learning images, and stores them in the storage device 103. When acquiring learning images (images of a size that can be acquired in one go using an SEM or the like), multiple images may be acquired using different sensors with the same field of view, and may be combined in the channel direction (for example, R, G, B, etc.) to be treated as a single learning image.

[0023] (ii) Step S101: If there is a previously trained segmentation engine 1011, the processor 101 loads that engine. If there is no previously trained segmentation engine 1011, the processor 101 can set random values ​​as parameters of the segmentation engine 1011.

[0024] (iii) Step S102 The processor 101 applies the segmentation engine 1011 to the group of training images to determine whether training is necessary. There are various methods for this determination, but for example, it may be determined that training is necessary if the average confidence level of the label map obtained when the segmentation engine 1011 is applied is equal to or less than a certain value. If the learning execution determination determines that learning is necessary (YES in step S102), the process proceeds to step S103. If the learning execution determination determines that learning is not necessary (NO in step S102), the process proceeds to step S104.

[0025] (iv) Step 103 The processor 101 performs a learning process to update the parameters of the segmentation engine 1011 .

[0026] (v) Step S105 The processor 101 captures an image of the sample at the location to be observed and acquires an image.

[0027] (vi) Step 106: The processor 101 acquires a label map using the segmentation engine 1011. The processor 101 repeats steps S104 and S105 to acquire images of all the locations to be observed and their label maps.

[0028] <Details of Learning Process (Step S103)> Fig. 2 is a flowchart for explaining the details of the learning process (step S103) of the segmentation engine 1011. In the flowcharts from Fig. 2 onwards, the process of each step and the data used in the process are shown together.

[0029] (i) Step S202 The segmentation engine 1011 performs a segmentation process on the training image 201 extracted (selected) from the training image group 200, and obtains a confidence map 203 and a label map 204. By performing the segmentation process, the training image 201 is classified (labeled) into image types for each pixel. Here, the confidence map is a map indicating the degree of certainty with which the pixels are labeled. For example, multiple labels are prepared in advance, and information indicating the certainty of classification for each label (confidence) is assigned. Furthermore, the label map is a map indicating the label with the highest confidence for each pixel.

[0030] There are various methods for obtaining a confidence map or label map using an image as input. For example, in segmentation using deep learning, the probability of each label is estimated for each pixel, and the label with the highest probability is used as the label map, and that probability is used as the confidence. Furthermore, in the case of the segmentation engine 1011 that performs estimation as a regression problem, one value is estimated for each pixel. In this case, for example, estimation may be performed multiple times by changing the values ​​of the image and parameters of the segmentation engine 1011, and calculation may be performed from the distribution of the obtained estimation results.

[0031] (ii) Step S205: The region R1 determination processing unit 1012 determines the region R1#206. The region R1#206 may be determined randomly from the learning image 201, or may be determined as a region with low confidence from the confidence map 203. In the latter case, this is to focus on learning the region with low confidence.

[0032] (iii) Step S207 The cutout processing unit 1013 cuts out the region R1#206 from the training image 201, thereby obtaining a training image 208 corresponding to the region R1#206, a confidence map 209 corresponding to the region R1#206, and a label map 210 corresponding to the region R1#206.

[0033] (iv) Step S211 The similar region search processing unit 1014 performs a similar region search process on the training image group 200 using the training image 208 of region R1#206, and searches the training image group 200 for a similar training image 213 including region R2#212 that is similar to the training image 208 of region R1#206.

[0034] (v) Step S214 The segmentation engine 1011 executes a segmentation process on the training similar images 213 acquired in step S211, and acquires a confidence map 215 and a label map 216 of the training similar images 213.

[0035] (vi) Step S217: The cutout processing unit 1013 cuts out the region corresponding to region R2#212 from the certainty map 215 and the label map 216 of the training similar image 213, and obtains the certainty map 218 corresponding to region R2#212 and the label map 219 corresponding to region R2#212.

[0036] (vii) Step S220: The learning necessity determination unit 1015 compares the confidence map 209 of the R1 region with the confidence map 218 of the R2 region to determine whether learning should be performed. Specifically, the learning necessity determination unit 1015 determines whether the confidence map 219 of R2 is higher than the confidence map 209 of the R1 region. If the confidence map 219 of R2 is higher than the confidence map 209 of the R1 region (YES in step S220), the process proceeds to step S221. If the confidence map 219 of R2 is lower than the confidence map 209 of the R1 region (NO in step S220), the process proceeds to step S202, where the segmentation process (step S202) and the cutout process (step S207) are performed using another learning image 201, and a new learning image 208, confidence map 209, and label map 210 for the R1 region are obtained.

[0037] (viii) Step S221: The segmentation engine update processing unit 1016 updates the segmentation engine so that the label map 210 of the R1 region approaches the label map 219 of the R2 region. This is done to improve the accuracy of the segmentation process. By repeating this learning, the result for the region R2#212 with a high degree of certainty is considered to be the correct answer, and it becomes possible to learn the region R1#206.

[0038] In the process according to Example 1, when the confidence map 218 for region R2 is lower than the confidence map 209 for region R1, the training of the segmentation engine 1011 may become unstable. Therefore, Example 2 proposes using low confidence search parameters 301 when determining the R1 region #206 and high confidence search parameters 302 to narrow down the candidates for region R2 #212 from the training image set 200. More specifically, the processing according to Example 2 includes: performing a segmentation process on the training image 201 using the segmentation engine 1011 to obtain a certainty; searching for region R1#206 from regions where the certainty is below a predetermined threshold (first threshold) by referring to a pre-prepared low-certainty search parameter (first threshold) 301, or from regions where the certainty is in the lower certain percentage; obtaining the certainty using the training image group 200 and the segmentation engine 1011; and searching for region R2#212 from regions where the certainty is above a predetermined threshold (second threshold) or from regions where the certainty is in the upper certain percentage by referring to a pre-prepared high-certainty search parameter (second threshold) 305.

[0039] <Improvement Scheme 1 of Learning Process> Fig. 3 is a flowchart illustrating improvement scheme 1 of the learning process (step S103) according to Example 2. Fig. 3 shows the process from the process after execution of step S202 in Fig. 2 to step S211 (steps S205 to S211 in Fig. 2 are replaced with the flowchart in Fig. 3).

[0040] (i) Step S302 When determining region R1#206 from the training image 201, its confidence map 203, and label map 204, the region R1 determination processing unit 1012 calculates a reference value using the low confidence search parameter 301 and determines region R1#206 from a region with low confidence. There are various methods for determining a region with low confidence, but for example, a method of determining a threshold (first threshold) in advance using the low confidence search parameter 301, or a method of regarding a region with a certain percentage of confidence as a region with low confidence based on the low confidence search parameter 301 and the confidence map 203 may be used.

[0041] (ii) Step 207 (same process as step S207 in Figure 2) The cutout processing unit 1013 cuts out region R1#206 from the training image 201, thereby obtaining a training image 208 corresponding to region R1#206, a confidence map 209 corresponding to region R1#206, and a label map 210 corresponding to region R1#206.

[0042] (iii) Step S303 The segmentation engine 1011 executes a segmentation process on each image in the training image set 200 to generate a confidence map set 304.

[0043] (iv) Step S306: The high confidence region search processing unit 2017 executes a high confidence region search process using the confidence map group 304 and the high confidence search parameters 305 to acquire high confidence regions, which are set as the region R2 candidate group 307. There are various methods for determining high confidence regions, but for example, a method can be used in which a threshold value (second threshold value) is determined in advance using the high confidence search parameters 305, or a method can be used in which regions with a certain percentage of high confidence levels are considered to be high confidence regions based on the high confidence search parameters 305 and the confidence map group 304.

[0044] (v) Step S211 The similar region search processing unit 1014 executes a similar region search process using the region R2 candidate group 307 and the training image 208 of region R1, and searches for a training similar image 213 that includes a region R2 similar to the training image 208 of region R1 from the region R2 candidate group 307. The subsequent processes (the processes from step S214 to step S211 in FIG. 2) are executed using this training similar image 213.

[0045] According to the second embodiment, it is possible to regard the result in the region R2 with a high degree of certainty as the correct answer, and to learn the region R1 with a low degree of certainty that is similar to the region R2.

[0046] In the learning process according to Example 1, it may be difficult to determine whether the label map obtained by the segmentation engine 1011 through learning is correct. Therefore, Example 3 proposes that the segmentation engine 1011 is configured with multiple feature extraction layers, and in the segmentation process, a region R2' in an input image where a circuit pattern similar to that of region R1' exists is calculated, and the success or failure of the segmentation is determined based on the segmentation result including the label map and confidence map corresponding to each of region R1' and region R2', or on features extracted from some of the feature extraction layers of the segmentation engine 1011. Note that the methods according to Examples 1 and 2 are applied to the learning process.

[0047] <Improvement of Segmentation Processing> FIG. 4 is a flowchart illustrating an improvement of the segmentation processing (details of step S105) according to the third embodiment.

[0048] (i) Step S401 The trained segmentation engine 1011 performs a segmentation process on the input search target image 400, and acquires features 402 (features obtained by deep learning) and a label map 403.

[0049] (ii) Step S402: In parallel with step S401 (or before or after step S401), the similar region search processing unit 1014 performs a similar region search process using the search target image 400, and acquires a region R2'#406 in the image in which a circuit pattern similar to that of region R1'#405 exists.

[0050] (iii) Step S411 The cutout processing unit 1013 cuts out and acquires the label map 407 and feature amount 408 corresponding to the region R1'#405, and the label map 409 and feature amount 410 corresponding to the region R2'.

[0051] (iv) Step S412: The success / failure determination processing unit 1018 compares the feature 408 with the feature 409 or the label map 407 with the label map 409 for the region R1'#405 and the region R2'#406 to determine whether they are similar. If the success / failure determination processing unit 1018 determines that they are similar, it determines that the segmentation process (step S401) was successful. If it determines that they are not similar, it determines that the segmentation process failed. This makes it possible to determine whether the segmentation was successful. There are various methods for determining the degree of similarity, and it may be determined based on, for example, the number of pixels with the same label map or the cosine similarity between the feature values.

[0052] This success / failure determination process makes it possible to detect whether the label map is incorrect. Various existing machine learning methods can be used for training the segmentation engine 1011, including deep neural networks. One method is the neural network described in Non-Patent Document 1. Specifically, a neural network called U-net, as shown in FIG. 5, can be used. Here, Y represents the input image, F1(Y), F2(Y), F3(Y), F4(Y), F5(Y), F6(Y), F7(Y), F8(Y), and F9(Y) represent feature values, and F(Y) represents the confidence level of each label. The label map is determined by searching for the label with the highest confidence level for each pixel. The feature values ​​and the final result are calculated using the following formulas, (Equation 1) to (Equation 10). Here, "*" represents a convolution operation, P represents an operation to apply a 2x2 max filter to an input image and reduce it by half in the spatial (XY) direction, UP represents an operation to upsample the input image to twice its size in the spatial direction, and CC represents an operation to combine two input images in the channel direction. W1 to W6 are convolution filters.

[0053] F1 (Y) = max (0, W1 * Y) ... (Formula 1) F2 (Y) = P (Y) ... (Formula 2) F3 (Y) = max (0, W2 * F2 (Y)) ... (Formula 3) F4 (Y) = P (F3 (Y)) ... (Formula 4) F5 (Y) = max (0, W3 * F4 (Y)) ...(Formula 5) F6(Y) = CC(UP(F5(Y)), F3(Y)) ...(Formula 6) F7(Y) = max(0, W4*F6(Y)) ...(Formula 9) F(Y)= max(0, W6*F9(Y))...(Formula 10)

[0054] The success / failure determination may be performed using one or more of the feature quantities obtained by the above processes. Alternatively, a feature quantity obtained by applying a convolution process to one or more of the feature quantities may be used.

[0055] In Examples 1 to 3, the label map of region R1 is trained to approximate the label map of region R2, which may result in the segmentation engine 1011 outputting a label map with a single label for all pixels for any training image. Therefore, Example 4 proposes that, in the training process, an image including a third partial image that is different (dissimilar) to the first partial image in the training image indicated by region R1#206 is searched for from the training image group 200, to acquire region R3 indicating the third partial image, and that training be performed so that the differences between the segmentation results of region R1 and region R3 or between the features extracted from some feature extraction layers of the segmentation engine 1011 become large.

[0056] <Improvement Scheme 2 of Learning Process> Fig. 6 is a flowchart for explaining improvement scheme 2 of the learning process (step S103) according to Example 4. Note that steps S601 to S607 shown in Fig. 6 are processes executed in parallel with steps S211 to S220 in Fig. 2 .

[0057] (i) Step S601 The dissimilar region search processing unit 1019 performs a dissimilar region search process on the training image 208 corresponding to region R1 and the training image group 200, and extracts a training dissimilar image 602) from the training image group 200 that includes region R3#603 that is dissimilar to the training image 208 corresponding to region R1.

[0058] (ii) Step S604 The segmentation engine 1011 executes a segmentation process on the training dissimilar images 602 to obtain a confidence map 605 and a label map 606.

[0059] (iii) Step S605 The cutout processing unit 1013 cuts out the confidence map 608 and label map 609 corresponding to the region R3#603 from the confidence map 605 and label map 606.

[0060] (iv) Step S211: The segmentation engine update processing unit 1016 updates the segmentation engine 1011 using the confidence map 209 and label map 210 corresponding to region R1 and the confidence map 608 and label map 609 corresponding to region R3 so that the segmentation results for region R1#206 and region R3#603 are different. This enables the segmentation engine 1011 to learn so that the label maps for region R1#206 and region R2#212 are closer to each other and so that the label maps for dissimilar regions R1#206 and region R3#603 are different from each other.

[0061] The fifth embodiment relates to the details of the similar region search process (step S211). In the fifth embodiment, in order to search for a region R2 that is visually similar to region R1, the similar region search process (step S211) proposes searching for region R2#212 by template matching using a first partial image in the training image indicated by region R1#206 as a template (training image 208).

[0062] <Details of Similar Region Search Process> FIG. 7 is a flowchart for explaining details of the similarity search process according to the fifth embodiment.

[0063] (i) Step S207 The cutout processing unit 1013 cuts out the region R1#206 of the learning image 201 and acquires a template (corresponding to the learning image 208 of the region R1).

[0064] (ii) Step S2111 The similar region search processing unit 1014 selects one image to be searched for a similar region from the training image group 200, and sets the selected image as a training similar image candidate 701. The similar region search processing unit 1014 then performs template matching using the template (training image 208), and sets the region with the highest similarity as a region R2 candidate 702.

[0065] (iii) Step S2112: The similar region search processing unit 1014 determines whether the similarity between the template 208 and the region R2 candidate 702 is equal to or greater than a certain value (a preset value). If the similarity is equal to or greater than the certain value, the similar region search processing unit 1014 outputs the region R2 candidate 702 to be determined as the training similar image 213 and region R2#212, and terminates the similar region search process. On the other hand, if the similarity is less than the certain value, the similar region search processing unit 1014 again searches for the region R2 candidate 702 using another training similar image candidate 701 from the training image group 200. If no region with a similarity equal to or greater than the certain value is found even after searching for the region R2 candidate 702 in all images in the training image group 200, the similar region search processing unit 1014 may consider the region R2 candidate 702 with the highest similarity to be region R2#213.

[0066] To capture and output high-quality images of semiconductor devices with circuit patterns formed on wafers using a scanning electron microscope or the like, with high visibility of defects and circuit patterns, it is necessary to understand the structure of the defects, circuit patterns, etc., such as their positions and shapes.

[0067] Therefore, in addition to the features proposed in Examples 1 to 5, Example 6 proposes configuring a label map to represent different labels depending on differences in the structure of semiconductor devices having circuit patterns formed on a wafer and the presence or absence of defects. Details of Example 6 will be described later using FIG. 8 . Example 6 relates to capturing an image of a device composed of multiple layers and estimating the pattern on the top layer, the pattern on the second layer, the pattern on the third layer, etc. from the image of the captured area. The layer on which a pattern resides can be determined through learning. First, in step S101 ( FIG. 1B ), the device to be processed is segmented using a segmentation engine 1011 trained on another device. Then, if it is determined in the learning execution determination process in step S102 ( FIG. 1B ) that accurate estimation was achieved, the learning process in step S103 is skipped. If it is determined that learning of the segmentation engine is necessary (if performance is poor), the learning process is continued to enable accurate estimation of device patterns.

[0068] <Label Map Based on Presence or Absence of Defect> FIG. 8 is a diagram for explaining generation of a label map based on the presence or absence of a defect according to the sixth embodiment.

[0069] The segmentation engine 1011 obtains a label map 801 that divides an image 801 of a semiconductor device having a circuit pattern formed on a wafer into, for example, a defect label 802, a label 803 of the circuit pattern in the first layer from the top, a label 804 of the circuit pattern in the second layer from the top, and a label 805 of the circuit pattern in the third layer from the top.

[0070] In order to obtain such a label map, for example, the relationship between the image and the ideal label map is learned in advance to construct a segmentation engine 1011, and the learned segmentation engine 1011 is read in the segmentation engine reading process (step S101).

[0071] In images of a semiconductor device having a circuit pattern formed on a wafer captured by a scanning electron microscope or the like, the appearance varies from image to image due to variations in the pattern shape, image noise, etc. Therefore, even if the features proposed in Examples 1 to 6 are used, there are cases where region R2 cannot be correctly searched for. In Example 7, in addition to at least one of the features of Examples 1 to 6, when searching for region R2 similar (similar in appearance) to region R1 in the similar region search process, design information of the semiconductor device is used for the search.

[0072] <Improvement Scheme 3 of Learning Process> Fig. 9 is a flowchart for explaining improvement scheme 3 of the learning process (step S103) of the segmentation engine 1011 according to Example 7. The flowchart in Fig. 9 replaces steps S202 to S216 of the flowchart in Fig. 2, and after step S909, steps S217 to S211 are executed.

[0073] (i) Steps S202 to S205 Description of these steps has been given in FIG. 2 and will therefore be omitted.

[0074] (ii) Step S902: The cutout processing unit 1013 references the design information 901 corresponding to the location where the learning image 201 was captured, cuts out an image corresponding to region R1# from the learning image 201 and region R1# 206 determined in step S205, and acquires design information 903, confidence map 209, and label map 210 corresponding to region R1 206. Here, the design information refers to layout data on the sample to be observed. For example, if the sample is a semiconductor, the design data is data in which edge information of the design shape of a semiconductor circuit pattern is written as coordinate data.

[0075] (iii) Step S904 The similar region search processing unit 1014 acquires design information 906 of a similar training image including a region R2 similar to the design information 903 corresponding to region R1#206 from the group of training images and the design information 904 corresponding to their imaging locations.

[0076] (iv) Step S907 The similar region search processing unit 1014 acquires a similar learning image 908 corresponding to the design information 906 of the similar learning image.

[0077] (v) Step S908 The segmentation engine 1011 executes segmentation processing on the training similar image 908 to obtain a confidence map 910 and a label map 911 corresponding to the region R2.

[0078] (vi) Steps S217 to S221 After step S908, steps S217 to S221 shown in FIG. 2 are executed using the confidence map 910 and label map 911 acquired in step S908.

[0079] Since the design information is not affected by changes in the pattern shape or image noise, the characteristics of the seventh embodiment make it possible to search for the region R2 without being affected by changes in the image.

[0080] Even if the features according to Examples 1 to 6 are used, if an image contains a defect and the defective portion and a normal portion are deemed to be similar, learning by the segmentation engine 1011 may fail. Therefore, Example 8 proposes that, in addition to at least one of the features proposed in Examples 1 to 6, in the similar region search process, a defect detection process is performed on the learning image group 200 to extract a defect region, and a region R2 is searched for based on the defect region.

[0081] <Improvement Scheme 4 of Learning Process> Fig. 10 is a flowchart for explaining improvement scheme 4 of the learning process (step S103) of the segmentation engine 1011 according to Example 8. In the flowchart of Fig. 10, the R1 region segmentation process of the flowchart of Fig. 2 is changed (step S1001 is added, and step S1002 is changed from step S205), and steps S1007 to S1009 are executed before step S211 of Fig. 2.

[0082] (i) Step S202 The segmentation engine 1011 performs a segmentation process on the learning image 201 including the defect 1000, and obtains a label map and a confidence map.

[0083] (ii) Step S1001: In parallel with step S202, or before or after step S202, the defect detection processing unit 1020 performs defect detection processing on the learning image 201 to detect defective areas. Defects can be detected by various methods, and for example, a method similar to the method disclosed in Patent Document 1 can be used.

[0084] (iii) Step S1002: The region R1 determination processing unit 1012 determines the region R1# 206. The region R1# 206 may be determined randomly from the learning image 201, or may be determined as a region with a low degree of certainty from the certainty map 203. Note that, in this example, a detected defect region is determined as the region R1# 206.

[0085] (iv) Step S207 The cutout processing unit 1013 cuts out the region R1#206 from the training image 201, thereby obtaining the training image 1003 corresponding to the region R1#206, the confidence map 1004 corresponding to the region R1#206, the label map 1005 corresponding to the region R1#206, and the defect detection result corresponding to the region R1#206 (defects are represented in white).

[0086] (v) Step S1007 The defect detection processing unit 1020 executes defect detection processing for each image in the learning image group 200, and generates a defect detection result group 1008.

[0087] (vi) Step S1008 The defect presence / absence determination processing unit 1021 executes a defect presence / absence determination process using the defect detection result 1006 for region R1 and the defect detection result group 1008, and acquires a region R2 candidate group 1010. In the defect presence / absence determination process, for example, if the defect detection result 1006 indicates the presence of a defect (if a defect is detected in region R1#206), the defect presence / absence determination processing unit 1021 may output the region determined to be defective in the defect detection result group 1008 as the region R2 candidate group 1010. If the defect detection result 1006 does not indicate the presence of a defect (if a defect is not detected in region R1#206), the defect presence / absence determination processing unit 1021 may output the region not determined to be defective in the defect detection result group 1008 as the region R2 candidate group 1010.

[0088] (vii) Step 211 The similar region search processing unit 1014 searches the region R2 candidate group 1010 for a region that is similar (similar in appearance) to the training image 1003 corresponding to region R1, and obtains the training similar image 213 and region R2#212.

[0089] (viii) Steps S214 to S221 Using the learning similar image 213 and region R2#212 acquired in step S211, steps S214 to S221 are executed in the same manner as in FIG.

[0090] The features of the eighth embodiment can reduce the possibility that a normal area and a defective area are determined to be similar areas, thereby improving the performance of similar area search based on defects.

[0091] In images of semiconductor devices with circuit patterns formed on wafers captured by a scanning electron microscope or the like, even similar circuit patterns often have different shapes due to minute roughness at the edges. Therefore, even if the features proposed in Examples 1 to 6 are used, the system may be trained to obtain the same label map for different roughness, etc. Example 9 proposes, in addition to at least one of the features of Examples 1 to 6, to accept as input in the segmentation engine update process a tolerance level for differences in circuit pattern shape between region R1#206 of the training image and region R2#212 of the similar training image, and calculate loss based on the tolerance level.

[0092] <Improvement Scheme 5 of Learning Process> Fig. 11 is a diagram showing an example of a change in a captured image. Fig. 12 is a flowchart for explaining improvement scheme 5 of the learning process (step S103) of the segmentation engine 1011 according to Example 9. In the flowchart of Fig. 12, after the process of step S202 is executed, a segmentation engine update process is performed taking into account the tolerance.

[0093] (i) Step S1205: In images of semiconductor devices with circuit patterns formed on wafers captured using a scanning electron microscope or the like, the pattern shape may vary even when similar patterns are captured. For example, as shown in FIG. 11 , there are image 1100 in which the roughness of the pattern shape has changed and image 1101 in which the corners of the pattern are rounded. In such cases, the label map 210 for region R1 and the label map 219 for region R2 do not necessarily match. Therefore, the segmentation engine update processing unit 1016 reads from the storage device 103 a tolerance 1201 indicating the degree of deviation between the label maps that is acceptable (whether the labels are considered similar). While referring to the tolerance, the segmentation engine update processing unit 1016 performs segmentation engine update processing using the confidence map 209 and label map 210 corresponding to region R1 and the confidence map 218 and label map 219 corresponding to region R2. An example of a method using tolerance in the segmentation engine update process is to treat the label map 219 as the same label map as the label map 210 corresponding to region R1 if the deviation of the boundary where the labels change in the label map 219 is within the tolerance.

[0094] The feature of the ninth embodiment makes it possible to train the segmentation engine 1011 even if the circuit pattern shapes in the region R1#206 and the region R2#212 differ to some extent.

[0095] In an image of a semiconductor device having a circuit pattern formed on a wafer captured by a scanning electron microscope or the like, even if the circuit patterns are similar, the appearance of the image changes due to changes in the pattern shape caused by effects such as roughness at the pattern edge, as well as noise, shading, and charging. Therefore, even if the features proposed in Examples 1 to 6 are used, it may not be possible to correctly search for region R2. Example 10 proposes, in addition to at least one of the features of Examples 1 to 6, inputting a tolerance level for one or more of differences in circuit pattern shape, image noise and shading, or brightness fluctuations due to charging in the similar region search process, and searching for region R2 (similar region search) based on the tolerance level.

[0096] <Improvement Scheme 6 of Learning Process> Fig. 13 is a flowchart for explaining improvement scheme 6 of the learning process (step S103) of the segmentation engine 1011 according to Example 10. In the flowchart of Fig. 13, a similar region is searched for in the similar region search process (step S1302) using a tolerance 1301. The other processes are the same as those in the flowchart of Fig. 2, and therefore will not be described again.

[0097] (i) Step S1205 In images of a semiconductor device having a circuit pattern formed on a wafer captured by a scanning electron microscope or the like, the pattern shape may change even when the same pattern is captured. For example, as shown in Figure 11, there are images 1100 in which the roughness of the pattern shape has changed and images 1101 in which the corners of the pattern are rounded. Furthermore, even when the circuit pattern does not change, there may be images 1102 affected by shot noise, 1103 affected by shading, and 1104 in which brightness fluctuations due to charging occur.

[0098] Therefore, the similar region search processing unit 1014 reads a parameter 1301 indicating the tolerance from the storage device 103 (or may accept the tolerance 1201 input by the operator). Based on the tolerance, the similar region search processing unit 1014 executes a similar region search process for the training image 208, confidence map 209, and label map 210 corresponding to region R1 and the training image group 200, taking into account changes in the image, to acquire a training similar image 213 and region R2# 212. There are various search processing methods that take image changes into account. For example, in cases where the pattern shape changes, such as in images 1100 and 1101, the tolerance can be set to indicate the maximum pixel change that is acceptable. For image 1102, which is affected by noise, noise removal processing can be applied in advance based on the tolerance. For image 1103, which is affected by shading, shading correction processing can be performed in advance based on the tolerance. For image 1104, which has brightness fluctuations due to charging, the maximum brightness change that is acceptable can be determined in advance based on the tolerance. The features of the tenth embodiment make it possible to search for the region R2#212 even if the appearance differs from image to image.

[0099] Even if the features proposed in Examples 1 to 10 are used, if there is an error in the label map for region R2 or if region R2 cannot be correctly searched, training of the segmentation engine may fail. Therefore, Example 11 proposes a GUI (Graphical User Interface) that checks the learning image group and label map corresponding to region R1 and region R2 and corrects either or both of the position of region R2 and the label map, in addition to at least one of the features of Examples 1 to 10.

[0100] <GUI for Correcting Region R2> FIG. 14 is a diagram showing an example of the configuration of a GUI 1400 for correcting at least one of the position of region R2 and the label map according to the eleventh embodiment.

[0101] The GUI 1400 displays an image 1401 corresponding to region R1 of the learning image, candidates 1402 and 1403 for region R2 obtained by the similar region search process, their label maps 1404 and 1405, and a target determination button 1406. Using the target determination button 1406, it is possible to select which label map to use as the label map for region R2. By reflecting the label map set here in the segmentation engine update process (step S221), it is possible to reflect the content selected or corrected for learning by the segmentation engine 1011. Furthermore, if all label maps are incorrect and no appropriate region R2 exists, the user may specify on the GUI a region that they believe is appropriate for region R2.

[0102] The features of the eleventh embodiment make it possible to select an appropriate region R2 and further correct the label map of the region R2. others

[0103] (i) The embodiments and examples of the present disclosure are not limited to the above-described contents and include various modifications. For example, although regions R1 and R2 are described using square regions, they can also be defined as regions of any shape. Furthermore, a region can be defined by a single pixel instead of multiple pixels.

[0104] (ii) The functions of this embodiment and each example can also be realized by software program code. In this case, a storage medium on which the program code is recorded is provided to a system or device, and the computer (or CPU or MPU) of that system or device reads the program code stored in the storage medium. In this case, the program code itself read from the storage medium realizes the functions of the above-mentioned embodiments, and the program code itself and the storage medium on which it is stored constitute the present disclosure. Examples of storage media for providing such program code include flexible disks, CD-ROMs, DVD-ROMs, hard disks, optical disks, magneto-optical disks, CD-Rs, magnetic tape, non-volatile memory cards, and ROMs.

[0105] Furthermore, an operating system (OS) running on a computer may perform some or all of the actual processing based on instructions in the program code, and the functions of the above-described embodiments may be realized by this processing.Furthermore, after the program code is read from a storage medium and written to memory on the computer, a CPU of the computer may perform some or all of the actual processing based on instructions in the program code, and the functions of the above-described embodiments may be realized by this processing.

[0106] Furthermore, the program code of the software that realizes the functions of the embodiments and each example may be distributed via a network and stored in a storage means such as a hard disk or memory of the system or device, or in a storage medium such as a CD-RW or CD-R, so that when used, the computer (or CPU or MPU) of the system or device reads and executes the program code stored in the storage means or storage medium.

[0107] The processes and techniques described herein are not inherently related to any specific device and can be implemented by a combination of components. Various types of general-purpose devices can also be added. A dedicated device may be constructed to perform the functions of this embodiment and each example. Various functions can also be formed by appropriately combining multiple components disclosed in this embodiment and each example. For example, some components may be omitted from all the components shown in the embodiment and each example, or components from different examples may be appropriately combined.

[0108] Although specific examples are described in this disclosure, they are in all respects for the purpose of explanation (understanding the technology of the present disclosure) and not for the purpose of limitation. Those skilled in the art will recognize that there are many combinations of hardware, software, and firmware suitable for implementing the technology of the present disclosure. For example, the software described can be implemented in a wide variety of programming or scripting languages, such as assembler, C / C++, Perl, Shell, PHP, Java (registered trademark), etc.

[0109] Furthermore, in the above-described embodiment, the control lines and information lines are those that are considered necessary for the explanation, and not all control lines and information lines in the product are necessarily shown. All components may be interconnected.

[0110] In addition, other implementations of the present disclosure will be apparent to those skilled in the art from consideration of the present embodiments and examples. The specification and examples are exemplary only, with the scope and spirit of the present disclosure being indicated by the following claims.

[0111] REFERENCE SIGNS LIST 1 Segmentation system 10 Segmentation device 20 Charged particle beam device 101 Processor (computer) 102 Communication device 103 Storage device 104 Input device 105 Output device 1011 Segmentation engine 1012 Region R1 determination processing unit 1013 Cutout processing unit 1014 Similar region search processing unit 1015 Learning necessity determination unit 1016 Segmentation engine update processing unit 1017 High confidence region search processing unit 1018 Success / failure determination processing unit 1019 Dissimilar region search processing unit 1020 Defect detection processing unit 1021 Defect presence / absence determination processing unit

Claims

1. An image segmentation method for assigning a desired label to each pixel of an input image by a computer, comprising: acquiring a group of training images including one or more images; using the group of training images to learn processing parameters of a segmentation engine that outputs a label map in which each pixel is labeled and a confidence map indicating the probability that each pixel belongs to each label; performing a segmentation process on the input image using the segmentation engine obtained by learning the processing parameters to obtain the label map of the input image, wherein learning the processing parameters of the segmentation engine includes: acquiring a training image from the group of training images and specifying a first region in the training image; acquiring a first partial image of the training image that corresponds to the first region; searching for a similar training image including a second partial image similar to the first partial image from images included in the group of training images, and acquiring a second region indicating the second partial image; An image segmentation method comprising: obtaining a label map and a confidence map corresponding to each of the first region and the second region using the segmentation engine, the training images, and the similar training images in the middle of training; and updating the processing parameters of the segmentation engine so that the label map corresponding to the first region approaches the label map corresponding to the second region.

2. The image segmentation method according to claim 1, wherein the computer updates the processing parameters of the segmentation engine when the confidence of the label map corresponding to the first region is lower than the confidence of the label map corresponding to the second region.

3. An image segmentation method according to claim 1, wherein specifying the first region in the training image includes: executing the segmentation process on the training image to obtain a certainty factor of the training image; and searching for the first region from a region where the certainty factor of the training image is equal to or less than a predetermined first threshold, or from a certain percentage of regions; and obtaining the second region in the similar training image includes: applying the segmentation engine to each image in the group of training images to execute a segmentation process to obtain a certainty factor of each image; and searching for the second region from a region where the certainty factor is equal to or greater than a predetermined second threshold, or from a certain percentage of regions.

4. An image segmentation method according to claim 1, wherein the segmentation engine is composed of a plurality of feature extraction layers, and obtaining a label map of the input image includes: searching for two regions in the input image that have similar patterns; and determining whether the segmentation process has been successful based on segmentation results including label maps and confidence maps corresponding to the two regions, or features extracted from some of the feature extraction layers of the segmentation engine.

5. An image segmentation method according to claim 1, wherein learning the processing parameters of the segmentation engine includes: searching the group of training images for an image including a third partial image dissimilar to the first partial image of the training image indicated in the first region; and learning the processing parameters so that there is a large difference between segmentation results including label maps and confidence maps corresponding to the third region indicating the third partial image and the first region, or between features extracted from a feature extraction layer of a part of the segmentation engine.

6. An image segmentation method as claimed in claim 1, wherein obtaining a second region representing the second partial image includes searching for the second region by a template matching process using the first partial image in a learning image corresponding to the first region as a template.

7. The image segmentation method according to claim 1, wherein the label map represents different labels depending on differences in the structure of a semiconductor device having a circuit pattern formed on a wafer and the presence or absence of defects.

8. The image segmentation method according to claim 7, wherein the computer uses design information of the semiconductor device when acquiring the second region.

9. An image segmentation method according to claim 7, wherein the computer extracts defect areas by performing defect detection processing on the group of learning images, generates a group of defect detection results consisting of images having defects, and searches for the second area based on the defect areas.

10. An image segmentation method as described in claim 7, wherein the computer determines whether the label map of the first region and the label map of the second region match based on information regarding the degree of tolerance regarding differences in circuit pattern shape between the first region of the training image and the second region of the similar training image.

11. An image segmentation method according to claim 7, wherein the computer searches for the second region based on information on the degree of tolerance for one or more of differences in the shape of the circuit pattern, image noise, shading, or brightness fluctuations due to charging.

12. An image segmentation method according to claim 1, further comprising: an image corresponding to the first region; a group of candidate images for the second region; and a label map corresponding to the group of candidate images for the second region; and displaying a GUI for correcting either or both of the position of the second region and its label map.

13. An image segmentation system that assigns a desired label to each pixel of an input image, comprising: a storage device that stores a program for training a segmentation engine and performing segmentation processing on an input image to be observed using the trained segmentation engine; and a computer that reads and executes the program from the storage device, wherein the computer performs the following steps: using a group of training images including one or more images, learning processing parameters for a segmentation engine that outputs a label map in which each pixel is labeled and a confidence map indicating the probability that each pixel belongs to each label; and using the segmentation engine obtained by learning the processing parameters to perform segmentation processing on the input image and obtain the label map for the input image, wherein the computer, in the process of training the processing parameters of the segmentation engine, performs the following steps: acquiring a training image from the group of training images and specifying a first region in the training image; and acquiring a first partial image of the training image that corresponds to the first region. An image segmentation system that performs the following processes: searching for similar training images including a second partial image similar to the first partial image from images included in the group of training images, and acquiring a second region indicating the second partial image; acquiring a label map and a confidence map corresponding to the first region and the second region, respectively, using the segmentation engine, the training images, and the similar training images in the middle of training; and updating the processing parameters of the segmentation engine so that the label map corresponding to the first region approaches the label map corresponding to the second region.

14. The image segmentation system according to claim 13, wherein the computer updates the processing parameters of the segmentation engine when the confidence of the label map corresponding to the first region is lower than the confidence of the label map corresponding to the second region.

15. An image segmentation system according to claim 13, wherein the segmentation engine is composed of multiple feature extraction layers, and the computer, when executing a process of acquiring a label map of the input image, executes the following processes: a process of searching for two regions having similar patterns in the input image; and a process of determining the success or failure of the segmentation process based on segmentation results including label maps and confidence maps corresponding to each of the two regions, or features extracted from some of the feature extraction layers of the segmentation engine.

Images