Non-conformity detection device and non-conformity detection method

Abstract

An incompatibility detection unit (10) comprises: an image conversion unit (12) that converts an input low-quality image (11) into a corresponding high-quality image (13) using a learning model (14); an incompatibility detection unit (15) that detects whether or not the input low-quality image (11) is incompatible with the learning model (14); an incompatibility reporting unit (16) that reports detected incompatibility; and a storage unit that stores, as a model-compatible region, the distribution of evaluation values of high-quality correct images (13B) used in the training stage of the learning model (14), in association with the learning model (14). The incompatibility detection unit (15) determines that the learning model (14) is incompatible when an evaluation value of the input low-quality image (11) is not within the model-compatible region.

Description

Technical Field

[0001] The present invention relates to a non-conformance detection device and a non-conformance detection method.

Background Art

[0002] A system for converting a low-quality image into a high-quality image using machine learning is described in Patent Document 1. In the system described in Patent Document 1, a learning model of machine learning is generated using a low-quality image and a high-quality image. Then, in the system described in Patent Document 1, this learning model is used to convert a low-quality input image into a high-quality output image by machine learning.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the system described in Patent Document 1, a learning model is prepared in advance for each purpose of image quality improvement. For example, if the purpose is noise removal, a learning model for noise removal corresponding to the magnitude of noise is prepared. Or, if it is for aberration improvement, a learning model corresponding to the magnitude of aberration is prepared. The user selects the learning model to be used based on the purpose of image quality improvement and the state of the image quality of the low-quality image (the situation of noise and aberration). And, in the system described in Patent Document 1, since the user visually selects a learning model for noise removal, it is easy to determine whether the selected learning model is effective for noise removal.

[0005] Furthermore, image transformation processing using a learned model has the property of transforming the input image to approximate the image data used during the learning model's training. Therefore, when a noisy input image is input to a learned model trained on noise-free learning data, it is expected that the output image will be de-decompressed to approximate the learning data.

[0006] On the other hand, unexpected side effects of image transformation processing, unrelated to the learning model's objectives, can also occur. For example, if the shape and position of subject A in the training data differ significantly from the shape and position of subject B in the input image, subject B in the output image may be transformed to resemble subject A. Such unexpected transformations can alter the meaning of the image, affecting its use in applications such as visually inspecting whether an item is normal or defective. This effect stems from the fact that the learning model used for image transformation processing is inherently unsuitable for the input image.

[0007] Therefore, the main objective of the present invention is to detect incompatibility in the learning model used for image conversion processing. [Means for solving the problem]

[0008] To solve the aforementioned problems, the non-conformity detection device of the present invention has the following features. This invention provides a learning model for inputting an image, performing image transformation, and then outputting the result. A non-conformity detection unit that detects whether the learning model is non-conformity or not, A non-conformity alarm unit that issues an alert for detected non-conformities, The system includes a storage unit that stores the distribution of evaluation values ​​of training images used in the training phase of the aforementioned learning model as a model-fitting region, in association with the learning model. The model fitting region is configured to indicate whether or not the evaluation value of the training image is within a certain range, with a constant width for quantizing the evaluation value of the training image. The non-conformity detection unit is characterized in that it determines that the learning model is unsuitable when the evaluation value of the image is not within the range of the model-fitting region. Other methods will be described later. [Effects of the Invention]

[0009] According to the present invention, it is possible to detect incompatibility in the learning model used for image conversion processing. [Brief explanation of the drawing]

[0010] [Figure 1] This is an external view showing a photograph of a semiconductor circuit pattern formed in two upper and lower layers according to this embodiment. [Figure 2] This is an explanatory diagram of the circuit pattern formed on the semiconductor wafer shown in Figure 1 according to this embodiment. [Figure 3] This is an explanatory diagram showing the amount of displacement to be determined from the semiconductor wafer in Figure 2 according to this embodiment. [Figure 4] This is a diagram showing the configuration of a non-conformity detection unit used in an image conversion system that converts low-quality images into high-quality images according to this embodiment. [Figure 5] This is a diagram showing the configuration of the non-conformity detection unit according to this embodiment. [Figure 6] This is a diagram showing the configuration of the model learning unit according to this embodiment. [Figure 7] This flowchart shows the processing flow of the non-conformity detection unit according to this embodiment. [Figure 8] This figure shows the method for defining the interval of the model-fitting region according to this embodiment. Two types of definition methods are illustrated below. [Figure 9] This figure shows an example of a method for setting the model fit region according to this embodiment. [Figure 10] This is a table showing a first example of a region representation method according to this embodiment. [Figure 11] This is a table showing a second example of a region representation method according to this embodiment. [Figure 12] This table shows a third example of a region representation method, extending the table in Figure 11 related to this embodiment to two-dimensional elements. [Figure 13]This is a table showing a modified example in which the directory name storing the learning pair images is stored in the cell value of the table in FIG. 12 related to this embodiment. [Figure 14] This is a data structure showing a fourth example of an area expression method obtained by expanding the table in FIG. 11 related to this embodiment into three-dimensional elements. [Figure 15] This is a configuration diagram of a non-conformance countermeasure section related to this embodiment. [Figure 16] This is a flowchart showing specific processing of non-conformance countermeasure processing related to this embodiment. [Figure 17] This is a configuration diagram showing a modified example of the non-conformance countermeasure section in FIG. 15 related to this embodiment. [Figure 18] This is a hardware configuration diagram of an image conversion system related to this embodiment. [Figure 19] This is an explanatory diagram showing how the numerical values in the table change by re-learning for a table of model conforming areas having the same expression method as in FIG. 11 related to this embodiment. [Figure 20] This is a graph showing the occurrence status of errors for a low-quality image before processing (before image quality improvement) by an image conversion section related to this embodiment. [Figure 21] This is a graph showing the occurrence status of errors for a high-quality equivalent image after processing (after image quality improvement) by an image conversion section related to this embodiment. [Figure 22] This is a diagram showing the state of measurement errors when image conversion is performed using a learning model in which the amount of deviation of learning pair images related to this embodiment is set as a model conforming area [-5, 5). [Figure 23] This is a graph showing the occurrence status of errors based on the results of the first learning related to this embodiment. [Figure 24] This is a graph showing the occurrence status of errors based on the results of the second learning related to this embodiment.

Mode for Carrying Out the Invention

[0011] Referring to the drawings, an image conversion system for converting a low-quality image according to this embodiment into a high-quality image will be described. [Examples]

[0012] Figure 1 is an external view showing a photograph of a semiconductor circuit pattern formed in two layers, upper and lower. In imaging environment 31A, the object of imaging is a semiconductor wafer in which the upper layer circuit 101 and the lower layer circuit 102 are formed into a multilayer structure (in this case, two layers, upper and lower) and integrated by etching, impurity doping, or thin film formation. The electron microscope 31 irradiates the semiconductor wafer with an electron beam (arrow in the figure) from above (from the side closer to the upper layer circuit 101) and takes an image. The semiconductor wafer in imaging environment 31A is a normal product in which no misalignment has occurred between the upper layer circuit 101 and the lower layer circuit 102.

[0013] In imaging environment 31B, the object of imaging is a semiconductor wafer in which the upper circuit 101 and lower circuit 102 are formed and integrated. In imaging environment 31B, the upper circuit 101 was formed slightly to the left of the lower circuit 102. Therefore, in imaging environment 31B, it is necessary to detect whether the semiconductor wafer is defective from the image captured by the electron microscope 31. Even in imaging environment 31C, the upper circuit 101 was formed with a slight misalignment to the right of the lower circuit 102. Therefore, even in imaging environment 31C, it is necessary to detect that the semiconductor wafer is defective from the image captured by the electron microscope 31.

[0014] Figure 2 is an explanatory diagram of the circuit pattern formed on the semiconductor wafer shown in Figure 1. The upper circuit 101 and lower circuit 102 have circuit patterns formed by using a mask for each layer. To make the explanation easier to understand, the circuit pattern 101p of the upper circuit 101 is made slightly longer vertically than the circuit pattern 102p of the lower circuit 102. In reality, numerous circuit patterns can be formed on a single semiconductor wafer, but for illustrative purposes, only a small number of circuit patterns are shown in Figure 2.

[0015] Figure 3 is an explanatory diagram showing the amount of displacement to be determined from the semiconductor wafer in Figure 2. Images 111-114 show a portion of the images of a semiconductor wafer in which the upper circuit 101 and lower circuit 102 have been formed and integrated (Figure 3 is a rough diagram, showing a magnified portion of the circuit pattern after superimposition). As shown in Figure 1, the electron beam of the electron microscope 31 passes through the upper circuit 101 on the near side and the lower circuit 102 on the far side, so both circuit patterns are captured in the image. For example, in captured images 111 to 114, the first circuit pattern (one of circuit patterns 102p), the second circuit pattern 101p (one of circuit patterns 101p), and the third circuit pattern 102p (one of circuit patterns 102p) are photographed side by side from left to right.

[0016] Images 111 and 112 are of a normal product with no positional misalignment, where the first, second, and third circuit patterns are aligned with equal distances d in the left-right direction. In other words, the amount of misalignment relative to this distance d is 0. Image 111 is a low-quality image containing not only the circuit pattern but also low resolution, white noise, and distortion (white noise is represented by hatching in the illustration). Image 112 is a high-resolution, high-quality image with minimal noise and distortion. Although it shows the same circuit pattern as image 111, it does not capture white noise. Here, a training pair of images is a pair of images that serve as teaching materials when training the learning model 14. One image in the pair becomes the input data to the learning model 14, and the other image in the pair becomes the output data to the learning model 14. For example, the learning model 14, which performs image transformation processing such as noise, will output a high-quality image like the captured image 112 when it receives a low-quality image like the captured image 111 as input. In this case, the pair of captured images 111 and 112, which depict the same subject, will be used as the training pair of images.

[0017] The following describes how to operate the trained model 14: The learning model 14 accepts the captured image 113 before image conversion as input data and outputs the captured image 114 after image conversion. Image 113 is a low-quality image of a defective product with misalignment. In addition to unwanted white noise, the distance d+10 between the second and third circuit patterns is larger than the distance d in image 111 (the second circuit pattern is shifted to the left by +10). Image 114 is a high-quality image obtained by applying the learning model 14 to image 113. As a result of the image quality improvement, unwanted white noise contained in image 113 has been neatly removed from image 114. However, as a side effect of the image quality improvement, the position of the circuit patterns in image 113 has been altered in image 114 to approximate the positional relationship of the circuit patterns in image 112 (the second circuit pattern has shifted to the left by an amount of +3).

[0018] In other words, as shown in the training pair images, it is expected that the amount of displacement will not change before and after image transformation (no error will occur). However, an error of 10-3=7 occurs between the displacement amount in captured image 113 = +10 and the displacement amount in captured image 114 = +3. As a result, the image classification results also suffer from the effects of this error, leading to the following misclassifications. Originally, based on the large deviation of "+10" in the captured image 113 (a deviation of threshold = 5 or more), it was possible to correctly determine that the captured semiconductor wafer was defective. However, based on a small deviation of "+3" in the captured image 114 (a deviation less than the threshold of 5), the system mistakenly identifies the captured semiconductor wafer as a normal product.

[0019] As explained above, when training the learning model 14 using captured images 112 with a small amount of displacement, the model learns not only image noise reduction but also circuit pattern arrangement information. Therefore, when captured images 113 with a large amount of displacement are input, it is thought that a process is performed to bring them closer to the arrangement information of captured images 112 learned during training, resulting in the output of captured image 114. Image noise reduction is an effective image transformation for improving measurement accuracy, but shifting circuit patterns like this is an inappropriate image transformation that reduces measurement accuracy. A learning model that performs such inappropriate image transformations is labeled as "unsuitable learning model."

[0020] The learning model mismatch cannot be detected by visually inspecting the captured image 114, image noise reduction is performed normally, and it is impossible to determine whether the circuit pattern movement is due to natural movement (shift) or movement caused by the learning model mismatch. Therefore, a mechanism is needed to detect the learning model mismatch and issue an alert. In other words, the captured image 114 itself has improved image quality due to the removal of white noise, making it difficult for the inspector to visually notice the error in the amount of displacement (change in the position of the circuit pattern) contained in the captured image 114. Therefore, the non-conformity detection unit 10 of this embodiment, which will be explained in Figure 4 and later, detects the error in the amount of displacement as a non-conformity of the learning model 14 and notifies the inspector of the detection result, thereby allowing the inspector to understand problems that would not be noticed by visual inspection of the captured image.

[0021] Figure 4 is a diagram illustrating the configuration of an image conversion system that converts low-quality images into high-quality images. The image conversion system comprises a non-conformity detection unit 10, a non-conformity countermeasure unit 20, an imaging device 30, an image utilization unit 40, and a control display unit 50. The non-conformity detection unit 10 detects non-conformities in the machine learning learning model 14, which is used in image conversion processing such as image quality improvement processing. The image conversion unit 12 of the non-conformity detection unit 10 converts the low-quality image 11 into a high-quality image using the learning model 14. The converted high-quality image is used for image observation and image measurement. Generally, the following shooting conditions are used to capture high-quality images. • Increase the imaging time. • Capture multiple short-exposure images and calculate their cumulative average. • Shine a strong light on it.

[0022] However, there are imaging devices 30 that cannot take images under such imaging conditions. For example, imaging devices 30 include electron microscopes 31 and X-ray tomography devices 32. An electron microscope 31 irradiates an object to be observed (for example, a semiconductor wafer) with an electron beam and observes the appearance of a circuit pattern formed on the wafer. Electron beam irradiation can damage circuit patterns, causing them to shrink. Shrinkage is caused by long exposure times (including capturing multiple short-exposure images) and high acceleration voltages. Therefore, high-quality images cannot be acquired at high frequency.

[0023] The same problems as those with the electron microscope 31 occur with the X-ray tomography apparatus 32. The X-ray tomography apparatus 32 takes images by irradiating the human body with X-rays. High-quality images can be obtained by irradiating for a long time or by increasing the X-ray intensity. However, this leads to an increase in X-ray exposure, making it difficult to obtain high-quality images using this method. Therefore, in order to minimize damage to the subject, it is better to obtain low-quality images 11. In this way, the low-quality images 11 captured by the imaging device 30 are stored in the image storage unit 33. The image conversion unit 12 then converts the low-quality images 11 in the image storage unit 33 into high-quality equivalent images 13 (images with image quality equivalent to high-quality images). Although this explanation describes a configuration in which the low-quality images 11 are input to the non-conformity detection unit 10 via the image storage unit 33, the images may also be input directly from the imaging device 30 to the non-conformity detection unit 10.

[0024] The high-quality equivalent image 13 output from the non-conformity detection unit 10 is input to the image utilization unit 40. The image utilization unit 40 includes an image observation processing unit 41, an image measurement processing unit 42, and an image classification processing unit 43. The image observation processing unit 41 performs various image processing operations, such as scaling, in order to observe the input image. The image measurement processing unit 42 measures the size of the shape using image processing. For example, the image measurement processing unit 42 performs image processing using the converted high-quality equivalent image 13 and extracts edge portions from the circuit pattern 101p of the upper circuit 101 and the circuit pattern 102p of the lower circuit 102 in Figure 2. Then, the image measurement processing unit 42 extracts the distance d between the edges shown in Figure 3.

[0025] The image classification processing unit 43 processes what kind of object the input image should be classified as. The image utilization unit 40 also performs image processing in areas where processing performance deteriorates with low-quality images 11, such as image segmentation processing which classifies regions of the image (though not shown in the diagram). The control display unit 50 displays various controls and processing results from the image utilization unit 40.

[0026] Figure 5 is a diagram showing the configuration of the non-conformity detection unit 10. The non-conformity detection unit 10 includes an image conversion unit 12 and a non-conformity detection unit 15. The non-conformity detection unit 10 stores a low-quality image 11 (captured image 113 in Figure 3), a high-quality equivalent image 13 (captured image 114 in Figure 3), and a learning model 14. The image conversion unit 12 outputs a high-quality equivalent image 13 from a low-quality image 11 captured by the input imaging device 30 using a learning model 14. The learning model 14 is a machine learning model such as a CNN (Convolutional Neural Network). The CNN is a means of converting the low-quality image 11 into a high-quality equivalent image 13 using the learning model 14.

[0027] The non-conformity detection unit 15 detects whether the low-quality image 11 input to be processed by the image conversion unit 12 is incompatible with the learning model 14. For information used by the non-conformity detection unit 15 to detect incompatibility, information regarding the model fit region, which indicates the evaluation values ​​(such as the amount of deviation) of the training pair images used during the learning process of the learning model 14, is registered in the storage unit of the non-conformity detection unit 10, associated with each learning model 14. The model fit region information is expressed, for example, as an interval of deviation between the training pair images used during learning. The non-conformity detection unit 15 determines that the learning model 14 is unsuitable when the evaluation value of the input low-quality image 11 is not within the range of the model-fitting region. The non-conformity notification unit 16 notifies the inspector, via a screen display, audio, or other means, that the learning model 14 used in the conversion process of the low-quality image 11, which was processed by the image conversion unit 12, is non-conformity, based on the detection results of the non-conformity detection unit 15.

[0028] Figure 6 is a diagram showing the configuration of the model learning unit 10B. The model learning unit 10B includes an image conversion unit 12 and a weight correction unit 12B. Model Learning Unit 10B It stores a low-quality image 11 (image 111 in Figure 3), a high-quality equivalent image 13 (image 112 in Figure 3), a high-quality ground truth image 13B, and a learning model 14. The high-quality ground truth image 13B is a high-quality image that serves as the target (ground truth) for improving the image quality of the low-quality image 11. For high-quality ground truth images 13B, in the case of semiconductor wafers, test wafers may be used, and in the case of X-rays, X-ray imaging phantoms simulating the human body may be utilized. High-quality ground truth image 13B is a training pair image that is paired with low-quality image 11, and is an image of the same position and field of view as low-quality image 11. Note that the conditions for capturing the high-quality ground truth image 13B involve increased electron beam and X-ray irradiation compared to the conditions for capturing the low-quality image 11. Once the learning model 14 is generated, the high-quality ground truth image 13B is not required when converting other low-quality images 11.

[0029] The weight correction unit 12B corrects the weights of the learning model 14 so that the image quality of the high-quality equivalent image 13 approaches that of the high-quality ground truth image 13B. The weights of the learning model 14 are, for example, the weight coefficients of a CNN network. Therefore, in the initial state, no weights are set for the learning model 14. Consequently, there is a discrepancy between the high-quality equivalent image 13, which is obtained by image conversion of the low-quality image 11 by the image conversion unit 12, and the high-quality ground truth image 13B. The weight correction unit 12B calculates a correction amount to correct the weight coefficients of the learning model 14 for correcting the amount of deviation, and modifies the learning model 14. The model learning unit 10B stores the distribution of evaluation values ​​of the training images used during the training phase of the learning model 14 as a model-fitted region in the memory unit, associating it with the learning model 14.

[0030] The weight correction process performed by the weight correction unit 12B is repeated using multiple training pair images and terminates when the amount of weight correction decreases. At the end of the weight correction process, the difference between the high-quality equivalent image 13 and the high-quality ground truth image 13B is minimized. The image conversion unit 12 uses the training model 14 at the end of the weight correction process to generate a high-quality equivalent image 13 with image quality close to that of the high-quality ground truth image 13B from the low-quality image 11.

[0031] Figure 7 is a flowchart showing the processing flow of the non-conformity detection unit 10. The image conversion unit 12 acquires the low-quality image 11 to be processed from the image acquisition storage unit 33 (multidimensional image acquisition data DB) (S11). Alternatively, the image conversion unit 12 may obtain the low-quality image 11 directly from the imaging device 30. The image conversion unit 12 uses the learning model 14 to obtain a high-quality equivalent image 13 from the acquired low-quality image 11 (image conversion) (S12). The image measurement processing unit 42 calculates the amount of displacement between the upper and lower layers by performing the image measurement processing described in Figure 4 using the acquired low-quality image 11 (S13).

[0032] The non-conformity detection unit 15 acquires information on the model-compatible regions registered in the DB of the learning model 14 (S14). The non-match detection unit 15 determines whether the amount of displacement of the low-quality image 11 calculated in S13 is within the model-fitting region of the learning model 14 (S15), and considers the learning model 14 outside the model-fitting region to be a learning model non-match. If the answer to S15 is Yes, the non-conformity detection unit 15 issues a non-conformity alert via the non-conformity alert unit 16 (S16). Furthermore, the non-conformity countermeasure unit 20 may execute countermeasures for the non-conformity (S17). On the other hand, if the answer to S15 is No, the deviation amount is output (S18).

[0033] The following is an example of the process for determining whether the learning model is unsuitable (S15). Figure 8 shows a diagram illustrating the method for defining the interval of the model-fitting region. Two definition methods are illustrated below. Interval definition method 121 defines the model fitting region as the interval of the displacement amounts of the training pair images used during training. In the example in Figure 8, the model fitting region is the union of the first interval ([-5, 5]) where the displacement amount is -5 to +5 and the second interval ([15, 20]) where the displacement amount is +15 to +20. In Figure 8, this is represented by a bar graph with hatched areas within the intervals.

[0034] Interval definition method 122 defines a model fitting region that takes into account the errors between the deviation amounts explained in Figure 3. To improve the reliability of the model-fit region, it is desirable to consider this error and narrow the boundaries of each interval (between the first and second intervals) inward. The amount of narrowing can be the standard deviation of the error, 3σ, where 3σ is the 99.7% confidence interval. In other words, the probability that a value near the boundary of the model-fit region defined by interval definition method 121 is outside the model-fit region defined by interval definition method 122 is 0.3% (=100-99.7). A model-fit region with increased reliability can be defined in this way.

[0035] Figure 9 shows an example of how to define the model fit region. This section explains how to define the interval for the model fit region. The horizontal axis of each graph (131, 132) represents the amount of deviation. Graph 131 shows the displacement values ​​of high-quality ground truth image 13B from a given pair of training images as black dots, and the influence curve of that displacement is shown as a bell curve. The function and tail width used to calculate the influence curve are determined through experiments, etc. There are multiple pairs of training images, and the distribution density for the displacement also differs. The vertical axis of Graph 131 shows the threshold for determining the influence of the displacement amount of the training pair images. If the influence curve is above this threshold, it is determined to be within the model fit region. In Figure 9, the first interval ([-35, -25]), the second interval ([-20, +5]), and the third interval ([+10, +25]) were determined to be intervals within the model fit region. Graph 132 shows the portion determined to be the model-fit region. When considering reliability, etc., the model-fit region can be reduced using the method described in interval definition method 122 in Figure 8, based on the model-fit region in Graph 132.

[0036] This section explains examples of how to represent data in a model-fitted domain (domain representation methods). Figure 10 is a table showing the first example of a region representation method. In this table, the values ​​of the model-fitting region obtained in Figure 9 are expressed as the amount of deviation between the start and end points. In this case, the learning model misfit determination process of the misfit detection unit 15 only needs to perform a determination process (S15) for each item number (#) to determine whether the amount of deviation of the low-quality image 11 falls within the model-fitting region.

[0037] Figure 11 is a table showing a second example of a region representation method. In this table, for each identifier of a model-fitting region (left column of the table), a flag (right column) is set indicating whether it is a model-fitting region (=1) or not (=0). The identifier for the model-fitted region is a single value, but it is treated as an identifier that identifies each model-fitted region with a width. For example, the identifier "-10" listed in the table represents the model-fitted region [-10, -5). The symbol [ includes the boundary, and the symbol ) does not. In this example, the quantization of the model-fitted region is set to 5, and the amount of shift at the starting point is used as its identifier. Furthermore, the model learning unit 10B can change the resolution of the model-fitted region by changing the quantization number.

[0038] When using the table in Figure 11, the non-match detection unit 15's learning model non-match determination process (S15) determines which model-matching region (left column of the table) the amount of deviation of the low-quality image 11 belongs to, and then refers to the flag in the right column of the table to which it belongs. If the flag is 1, it is a model-matching region, and if it is 0, it is not a model-matching region. For example, if the displacement amount of the low-quality image 11 is 16, the non-conformity detection unit 15 determines that it is included in the model-fitting region [15, 20)] and assigns the region identifier to "15". Then, the non-conformity detection unit 15 determines that the model is non-conformity by referring to the region identifier "15" → flag "0" from the table in Figure 11.

[0039] Furthermore, the representation method shown in Figure 11 is easily multidimensionalized. Until now, only the lateral displacement was considered within the model fitting domain. In reality, vertical displacement also occurs, as well as other factors completely unrelated to displacement, such as changes in image quality due to differences in the acceleration voltage of the electron microscope 31. It is necessary to perform training using training pair images for each element.

[0040] Figure 12 is a table showing a third example of a region representation method, extending the table in Figure 11 to two-dimensional elements. Note that while the identifier range in the table in Figure 11 was from -10 to 40, the identifier range in the table in Figure 12 is from -20 to 30. In the table in Figure 12, the horizontal columns indicate identifiers for the model-fit region representing the horizontal displacement, and the vertical columns indicate identifiers for the model-fit region representing the vertical displacement. The numbers inside the cells where the horizontal and vertical columns intersect represent the model-fit region as 1 and not as 0. In this example, the horizontal displacement of [0, 20) indicates the model-fit region, and the vertical displacement of [0, 10) indicates the model-fit region. When using the table in Figure 12, the learning model mismatch determination process (S15) of the mismatch detection unit 15 only needs to determine which cell in the table corresponds to the amount of misalignment of each element of the low-quality image 11, and then refer to the numerical value inside that cell.

[0041] Figure 13 shows a modified version of the table in Figure 12, where the cell values ​​contain the directory names where the training pair images are stored. The string inside the cell is represented as 0 if it is not in the model fitting region, and a non-zero value if it is in the model fitting region.

[0042] Figure 14 shows a data structure representing a fourth example of a region representation method, which extends the table in Figure 11 to three-dimensional elements. In Figure 14, the X-axis represents the lateral displacement, the Y-axis represents the vertical displacement, and the Z-axis represents the acceleration voltage. While representations of three dimensions or more cannot be shown, the domain representation method can be extended up to n dimensions. When using the data structure shown in Figure 14, the learning model mismatch determination process (S15) of the mismatch detection unit 15 only needs to determine which cell in the table the combination of the horizontal displacement, vertical displacement, and acceleration voltage of the low-quality image 11 belongs to, and then refer to the numerical value (not shown) inside the cell to which it belongs. According to Example 1, it becomes possible to detect when the learning model is unsuitable for the image being processed, and to issue a warning when the measurement result has a large error.

[0043] Figure 15 is a diagram showing the configuration of the non-conformity countermeasure unit 20. The non-conformity countermeasure unit 20 includes a countermeasure method search unit 22, a countermeasure method suggestion unit 23, a retraining data input unit 24, a model change unit 25, an existing model retraining unit 26, and a new model learning unit 27. The non-conformity countermeasure unit 20 stores a countermeasure method DB 21. The details of the non-conformity countermeasure unit 20 will be described below with reference to Figure 16.

[0044] Figure 16 is a flowchart showing the specific procedures for corrective action against nonconformities (S17). This flowchart first explains a countermeasure process that does not involve the operator, and then describes a modified version that involves the operator. The model change unit 25 searches for another model that can process the amount of deviation (S171). The model change unit 25 determines whether the search in S171 was successful and whether another model was found (S172). If the answer in S172 is Yes, the model change unit 25 changes the model used to the other model found from the currently unsuitable model (S173). The model used is the learning model 14 that the image conversion unit 12 uses for conversion processing. In other words, when the non-conformity detection unit 15 detects a non-conformity, the model change unit 25 searches the storage unit of the non-conformity countermeasure unit 20 for another learning model 14 that corresponds to a model conformity region that matches the evaluation value of the input low-quality image 11, and controls the image conversion unit 12 to use the other learning model 14 for the conversion process of the input low-quality image 11. As a result, the image conversion unit 12 can perform appropriate (less error-free) image conversion processing based on the learning model 14 that was modified in S173 to fit the low-quality image 11 to be processed.

[0045] If the answer in S172 is No, the countermeasure search unit 22 retrieves an existing learning model that can be retrained from the learning model 14 DB (S174). The model retrieved in S174 may be the currently unsuitable model or another existing learning model. The countermeasure search unit 22 determines whether it succeeded in retrieving the existing learning model in S174 (S175). If the answer in S175 is Yes, the existing model retraining unit 26 retrains the existing model (S176). In other words, if the non-conformity detection unit 15 detects a non-conformity, the existing model retraining unit 26 expands the model fit region of the learning model 14 by retraining the non-conformity learning model 14 based on additional high-quality ground truth images 13B. If the answer in S175 is No, the new model learning unit 27 learns the new model (S177).

[0046] Therefore, the retraining data input unit 24 obtains high-quality images of training data to be used in the retraining process (S176) of the existing model retraining unit 26 and the training process (S177) of the new model training unit 27. An existing model is a model that has undergone training at least once, and its model fit region includes one or more intervals. On the other hand, a new model is an initial model that has not undergone training, and its model fit region does not include any intervals.

[0047] Figure 17 is a configuration diagram showing a modified example of the non-conformity countermeasure unit 20 of Figure 15. In Figure 17, a retraining data acquisition unit 24b is provided instead of a retraining data input unit 24. The retraining data acquisition unit 24b obtains high-quality images without operator intervention by performing automatic operation according to the programmed operating procedure of the imaging device 30. In other words, the retraining data acquisition unit 24b accepts the input of additional high-quality ground truth images 13B that have been captured by operating the imaging device 30 according to a pre-set operating procedure. The retraining data acquisition unit 24b may also perform the other processes shown in Figure 16 in an automated manner.

[0048] Furthermore, the countermeasure presentation unit 23 may present the user with the following three types of countermeasures found by the countermeasure search unit 22 from the countermeasure method DB 21, as well as the work procedures required for each countermeasure, allowing the user to select which countermeasure to adopt. For this reason, the countermeasure method DB 21 contains information indicating what work should be done next if the non-conformity notification unit 16 determines that the model is non-conformity. (Countermeasure 1) Change the model used by the model change unit 25 (S173). The countermeasure presentation unit 23 displays candidates for the model to be changed, and the operator may confirm a candidate for the model used by pressing a confirmation button or the like.

[0049] (Countermeasure Method 2) Retraining of the existing model by the existing model retraining unit 26 (S176, details explained in Example 2). Note that when retraining is performed, it is necessary to capture high-quality images as training data. Capturing high-quality images must be done using a different procedure than normal imaging (capturing low-quality images 11). Therefore, the countermeasure method presentation unit 23 may display this procedure (recipe) on the screen in an easy-to-understand manner for the operator and provide support for the operation. For this reason, the countermeasure method DB 21 stores, for example, a method for capturing additional high-quality ground truth images 13B for retraining, such as "To capture high-quality images, shine a strong light on the subject." The contents stored in this countermeasure method DB 21 are various messages to be displayed on the operator's screen. (Countermeasure Method 3) Training of the new model by the new model learning unit 27 (S177). Similar to (Countermeasure Method 2), the countermeasure method presentation unit 23 may display the work procedure (recipe) on the screen in an easy-to-understand manner for the operator.

[0050] Figure 18 is a hardware configuration diagram of the image conversion system. Each processing unit of the image conversion system (non-conformity detection unit 10, non-conformity notification unit 16, non-conformity countermeasure unit 20, image utilization unit 40, and control display unit 50) is configured as a computer 900 having a CPU 901, RAM 902, ROM 903, HDD 904, communication I / F 905, input / output I / F 906, and media I / F 907. The HDD 904 is, for example, a storage device that stores the learning model 14. The communication interface 905 is connected to an external communication device 915. The input / output interface 906 is connected to the input / output device 916. The media interface 907 reads and writes data to the recording medium 917. Furthermore, the CPU 901 controls each processing unit by executing a program (also called an application or app) loaded into the RAM 902. This program can also be distributed via a communication line or by recording it on a recording medium 917 such as a CD-ROM. Furthermore, each processing unit in the image conversion system can be any hardware capable of image processing. For example, it can be a computer equipped with a CPU or GPU, a storage device such as an HDD, or a field-programmable gate array (FPGA) that can program logic circuits, or even custom-built hardware. [Examples]

[0051] In Example 2, the details of the retraining process (S176) of the existing model by the existing model retraining unit 26 will be explained. As an example of the retraining process, a method for adjusting the learning model using fine tuning, which is commonly used, and expanding the model fitting region while fitting the learning model will be described. In fine tuning, the existing model is used as the state before training. The training pair images used for training are both the training pair images used when the existing model was created (registered in the model fitting region) and the training pair images of the region to be expanded in the model fitting region (before registration in the model fitting region).

[0052] By using both training pair images, a training model is generated that fits both the model-fitting region covered by the initial training model and the additional model-fitting region. Since the extended region also requires training pair images, high-quality ground truth images 13B are needed in addition to the low-quality images 11. Therefore, the acquisition of high-quality ground truth images 13B is required. Fine-tuning allows for significantly faster model generation compared to training a model with no prior training, and is effective in improving processing throughput. Furthermore, fine-tuning is also suitable for creating learning models that combine multiple objectives, such as using a pre-trained model for noise reduction as the initial state and then training it using training pair images for aberration correction. In other words, the retrained learning model can perform image transformation processes that improve both noise reduction and aberration correction, for example.

[0053] Figure 19 is an explanatory diagram showing how the numerical values ​​in the model fitting region table, which uses the same representation method as in Figure 11, change during retraining. The table in Figure 19 combines the initial learning model (columns 1-3), the model showing the results of the first retraining (columns 4-5), and the model showing the results of the second retraining (columns 6-7) into a single table. Since the model fit region identifier is the same for all three models, it is listed only in column 1. The 2nd, 4th, and 6th columns of the "Judgment" column are flags indicating whether the region is a model fit (=1) or not (=0). Columns 3, 5, and 7 of the "Storage Location" column indicate the directory name where the training image pairs used for fine-tuning and adjusting the learning model are stored. The training image pairs in this directory will be used as training data during retraining.

[0054] The initial trained model shows the model fit region [-5, 5)], the results of the first training run are expanded to the model fit region [-15, 5)], and the results of the second training run are expanded to the model fit region [-40, 5)]. The training pair images used in the first training run are stored in directories named "A003" and "A004", respectively. The training pair images used in the second training run are stored in directories named "A005" through "A009", respectively. For example, if the displacement of the newly processed low-quality image 11 is in the region [-15, -5), the initial trained model will be incompatible with some intervals [-15, -5] because they are outside the interval. However, using the results of the first training run will result in a complete fit within the interval.

[0055] Figure 20 is a graph showing the error situation for the low-quality image 11 before processing by the image conversion unit 12 (before image quality improvement). Figure 21 is a graph showing the occurrence of errors in the high-quality equivalent image 13 after processing (image quality improvement) by the image conversion unit 12. The horizontal axis of the graph shows the amount of displacement obtained using the high-quality equivalent image 13. The vertical axis of the graph shows the error between the amount of displacement obtained using the low-quality image 11 or the high-quality equivalent image 13 and the amount of displacement obtained using the high-quality ground truth image 13B. The learning model 14 was created using training pair images with upper and lower layer displacements ranging from -40 to +5.

[0056] 200 images were used to generate the graphs in Figures 20 and 21. Figure 21 shows that the error variability is smaller in the total deviation amount compared to Figure 20. Smaller error variability indicates improved accuracy, and it can be seen that the image conversion unit 12 has improved the measurement accuracy. The reason the horizontal axis is the deviation amount obtained using high-quality equivalent images 13, rather than the deviation amount obtained from high-quality images, is that high-quality images cannot be used in the learning model misfit determination process for the reasons described in the problem of obtaining high-quality images. In Figure 21, a learning model was created using training pair images with upper-lower layer misalignments ranging from -40 to +5. Upper-lower layer misalignments occur due to some manufacturing process issue, and in actual operation, it is difficult to prepare training pair images with such a wide range of misalignments.

[0057] Figure 22 shows the measurement error when image transformation is performed using a trained model in which the displacement amount of the training pair images is set as the model fitting region [-5, 5)]. In the model-fitting region shown in Figure 22, the error variability is approximately the same as in Figure 21, indicating an improvement in measurement accuracy. However, for images shifted to the negative side of the model-fitting region, the error increases as the distance from the model-fitting region increases. Thus, if the amount of shift of the low-quality image 11 is included in the model-fitting region, the error variability is small and the inappropriate movement of the circuit pattern is also small. If the amount of shift of the low-quality image 11 is not included in the model-fitting region, the error variability is large and the inappropriate movement of the circuit pattern is also large.

[0058] Figure 23 is a graph showing the occurrence of errors based on the results of the first training run. Through fine-tuning, the model fit region has been expanded to the interval [-15, 5), and it can be seen that the error (vertical axis) within this interval has decreased. Based on this model fit region [-15, 5)] of the learned model 14, the non-fit detection unit 15 determines the subsequent model fit.

[0059] Figure 24 is a graph showing the occurrence of errors based on the results of the second training run. Through fine-tuning, the model fit region has been expanded to the interval [-40, 5), and it can be seen that the error (vertical axis) has decreased in almost the entire interval. Based on this model fit region [-40, 5)] of the learned model 14, the non-fit detection unit 15 determines the subsequent model fit.

[0060] As shown in Figure 9, the amount of misalignment in the acquired training pair images can be concentrated in one area or become coarse. If many images are concentrated in the identifier region of a certain model fitting region, the weight of that region may become large, resulting in unbalanced retraining. Therefore, it is desirable to keep the quantization width of the model-fit region in the table representing the model-fit region, such as in Figure 11, constant. This prevents such imbalances from occurring by keeping the number of training pair images constant for each model-fit region.

[0061] In the non-conformity detection unit 10 of this embodiment described above, the non-conformity detection unit 15 determines whether the learning model 14 used to convert the low-quality image 11 to a high-quality equivalent image 13 is suitable for the low-quality image 11. Based on the determination of the non-conformity detection unit 15, the non-conformity notification unit 16 notifies that the learning model 14 is unsuitable for the low-quality image 11 of the input data, and therefore the high-quality equivalent image 13 is not the desired high-quality image. This allows the user to view high-quality equivalent images 13 as images. Improvements have been made to it. Even if this is the case, it is possible to identify model incompatibility, thus preventing misjudgments of image content based on high-quality equivalent images 13.

[0062] Furthermore, the non-conformity countermeasures unit 20 takes measures such as adding training target images corresponding to the low-quality images 11 of the input data that the learning model 14 has determined to be non-conformity to the training data and performing retraining. This creates a learning model 14 that is compatible with the low-quality images 11. In this way, by retraining the learning model 14, the model fit domain corresponding to that learning model 14 can be sequentially expanded. For example, if the learning model is fit with respect to noise but not with respect to aberrations, the existing model retraining unit 26 can generate a learning model 14 that is fit with respect to both noise and aberrations by retraining using both noise and aberration images.

[0063] It should be noted that the present invention is not limited to the embodiments described above, and various modifications are included. For example, the embodiments described above are described in detail to make the present invention easier to understand, and are not necessarily limited to those having all the configurations described. Furthermore, it is possible to replace parts of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. Furthermore, it is possible to add, delete, or replace some of the configurations in each embodiment with other configurations. In addition, some or all of the above configurations, functions, processing units, processing means, etc., may be implemented in hardware, for example, by designing them as integrated circuits. Furthermore, each of the aforementioned configurations and functions may be implemented in software by the processor interpreting and executing programs that realize each of these functions.

[0064] The information such as programs, tables, and files that implement each function can be stored in memory, storage devices such as hard disks and SSDs (Solid State Drives), or recording media such as IC (Integrated Circuit) cards, SD cards, and DVDs (Digital Versatile Discs). Cloud computing can also be utilized. Furthermore, the control lines and information lines shown are those deemed necessary for explanatory purposes, and not all control lines and information lines are necessarily shown in the actual product. In reality, it can be assumed that almost all components are interconnected. Furthermore, the communication method connecting each device is not limited to wireless LAN; it may be changed to wired LAN or other communication methods. [Explanation of symbols]

[0065] 10. Non-conformity detection unit (non-conformity detection device) 10B Model Learning Section 11. Low-quality image (image before conversion) 12 Image conversion unit 12B Weight Correction Unit 13. High-quality equivalent image (converted image) 13B High-quality correct images (training images) 14 Learning Models 15 Non-conformity detection unit 16 Non-conformity notification unit 20. Non-conformity countermeasures unit (non-conformity detection device) 21 Countermeasures Database 22 Countermeasures Search Unit 23. Section for Presenting Countermeasures 24 Retraining Data Input Section 24b Retraining Data Collection Unit 25. Changes to the model used 26 Existing Model Retraining Unit 27 New Model Learning Department 30 Imaging device 31 Electron Microscope 32 X-ray Tomography Equipment 33 Image storage unit 40 Image Utilization Section 41 Image observation processing unit 42 Image Measurement Processing Unit 43 Image Classification Processing Unit 50 Control display unit

Claims

1. A learning model for taking an image as input, performing image transformation, and then outputting the result, A non-conformity detection unit that detects whether the learning model is non-conformity or not, A non-conformity alarm unit that issues an alert for detected non-conformities, The system includes a storage unit that stores the distribution of evaluation values ​​of training images used in the training phase of the aforementioned learning model as a model-fitting region, in association with the learning model. The model fitting region is configured to indicate whether or not the evaluation value of the training image is within a certain range, with a constant width for quantizing the evaluation value of the training image. The non-conformity detection unit is characterized in that it determines the learning model is non-conformity when the evaluation value of the image is not within the range of the model-conformity region. Non-conformity detection device.

2. The aforementioned non-conformity detection device further includes a model change unit, The aforementioned model change unit, when the non-conformity detection unit detects a non-conformity, searches the storage unit for another learning model corresponding to the model-fitting region that matches the evaluation value of the image, and uses the other learning model for the image conversion process. The non-conformity detection device according to claim 1.

3. The aforementioned non-conformity detection device further includes an existing model retraining unit, The aforementioned existing model retraining unit is characterized in that, when the non-fit detection unit detects a non-fit, it retrains the non-fitting learning model based on the additional learning images, thereby expanding the model fit region of the learning model. The non-conformity detection device according to claim 1.

4. The aforementioned non-conformity detection device further includes a countermeasure method suggestion unit and a retraining data input unit. The countermeasure method presentation unit presents a countermeasure method for non-conformity, including a method for capturing additional learning images. The retraining data input unit is characterized by receiving input of additional training images captured using the presented shooting method. The non-conformity detection device according to claim 3.

5. The aforementioned non-conformity detection device further includes a retraining data acquisition unit. The retraining data acquisition unit is characterized by receiving input of additional training images that have been captured by operating the imaging device according to a pre-set operating procedure. The non-conformity detection device according to claim 3.

6. The non-conformity detection device comprises a learning model, a non-conformity detection unit, a non-conformity alarm unit, and a storage unit. The aforementioned learning model is a model that takes an image as input, performs image transformation, and then outputs the result. The non-conformity detection unit performs the step of detecting whether or not the learning model is non-conformity, The aforementioned non-conformity alarm unit performs the step of alarming the detected non-conformity, The memory unit stores the distribution of evaluation values ​​of the training images used during the training phase of the learning model as a model-fitting region, associated with the learning model. The model fitting region is configured to indicate whether or not the evaluation value of the training image is within a certain range, with a constant width for quantizing the evaluation value of the training image. The non-fit detection unit is characterized by performing the step of determining that the learning model is non-fit when the evaluation value of the image is not within the range of the model-fit region. Non-conformity detection method.

7. The aforementioned image is an image of a circuit pattern formed on a semiconductor wafer, The evaluation value of the aforementioned image is characterized by being a quantitative indicator related to the observed object or shooting conditions, such as the amount of displacement, aberration, or noise reduction of the circuit pattern obtained from the aforementioned image. The non-conformity detection device according to claim 1.

Images