Incompatibility detection device and incompatibility detection method

Abstract

An incompatibility detection unit (10) comprises: an image conversion unit (12) that converts an input low-quality image (11) into a corresponding high-quality image (13) using a learning model (14); an incompatibility detection unit (15) that detects whether or not the input low-quality image (11) is incompatible with the learning model (14); an incompatibility reporting unit (16) that reports detected incompatibility; and a storage unit that stores, as a model-compatible region, the distribution of evaluation values of high-quality correct images (13B) used in the training stage of the learning model (14), in association with the learning model (14). The incompatibility detection unit (15) determines that the learning model (14) is incompatible when an evaluation value of the input low-quality image (11) is not within the model-compatible region.

Description

Non-conformity detection device and non-conformity detection method

[0001] The present invention relates to a mismatch detection device and a mismatch detection method.

[0002] Patent Literature 1 describes a system that converts low-quality images into high-quality images using machine learning. In the system described in Patent Literature 1, a machine learning learning model is generated using low-quality images and high-quality images. Then, in the system described in Patent Literature 1, this learning model is used to convert low-quality input images into high-quality output images through machine learning.

[0003] International Publication No. 2021 / 095256

[0004] In the system described in Patent Document 1, a learning model is prepared in advance for each purpose of image quality improvement. For example, if the purpose is noise removal, a learning model for noise removal corresponding to the magnitude of the noise is prepared. Or, if the purpose is aberration improvement, a learning model corresponding to the magnitude of the aberration is prepared. The user selects the learning model to use based on the purpose of image quality improvement and the image quality state of the low-quality image (noise and aberration conditions). In the system described in Patent Document 1, the user visually selects the learning model for noise removal, making it easy to determine whether the selected learning model is effective for noise removal.

[0005] Note that image conversion processing using a learning model has the property of converting an input image so as to approximate the image of the learning data used when the learning model was trained. Therefore, when a noisy input image is input to a learning model trained from noise-free learning data, it is expected that an output image in which noise has been removed from the input image so as to approximate the learning data will be obtained.

[0006] On the other hand, unexpected side effects of image transformation processes unrelated to the purpose of the learning model may also occur. For example, if the shape and position of subject A in the learning data differ significantly from the shape and position of subject B in the input image, subject B in the output image will be deformed to resemble subject A. Such unexpected deformations change the meaning of the image, affecting the use of the converted image, for example, when visually inspecting whether the subject is normal or defective. This effect is caused by the learning model used in the image transformation process being incompatible with the input image.

[0007] Therefore, the main object of the present invention is to detect incompatibility of the learning model used in image conversion processing.

[0008] In order to solve the above problems, the non-conformity detection device of the present invention has the following features: an image conversion unit that converts an input pre-conversion image into a post-conversion image using a learning model; a non-conformity detection unit that detects whether the pre-conversion image and the learning model are incompatible; a non-conformity reporting unit that reports the detected non-conformity; and a storage unit that stores the distribution of evaluation values ​​of training images used in the training stage of the learning model as a model suitability region in association with the learning model, and the non-conformity detection unit determines that the learning model is incompatible when the evaluation value of the pre-conversion image is not within the model suitability region. Other means will be described later.

[0009] According to the present invention, it is possible to detect incompatibility of a learning model used in image conversion processing.

[0010] 1 is an external view showing the imaging of a semiconductor circuit pattern formed on two upper and lower layers according to this embodiment. FIG. 1 is an explanatory diagram of a circuit pattern formed on the semiconductor wafer of FIG. 1 according to this embodiment. FIG. 2 is an explanatory diagram showing a deviation amount determined from the semiconductor wafer of FIG. 2 according to this embodiment. FIG. 3 is a configuration diagram of a non-conformance detection unit used in an image conversion system for converting low-quality images into high-quality images according to this embodiment. FIG. 4 is a configuration diagram of a non-conformance detection unit according to this embodiment. FIG. 5 is a configuration diagram of a model learning unit according to this embodiment. FIG. 6 is a flowchart showing the processing flow of the non-conformance detection unit according to this embodiment. FIG. 7 is a diagram showing a method for defining a section of a model-fitting region according to this embodiment. Two types of definition methods are illustrated below. FIG. 8 is a diagram showing an example of a method for setting a model-fitting region according to this embodiment. FIG. 9 is a table showing a first example of a region representation method according to this embodiment. FIG. 10 is a table showing a second example of a region representation method according to this embodiment. FIG. 11 is a table showing a third example of a region representation method in which the table of FIG. 11 according to this embodiment is expanded to two-dimensional elements. FIG. 12 is a table showing a modification in which directory names in which paired learning images are stored are stored in the cell values ​​of the table of FIG. 12 according to this embodiment. FIG. 13 is a data structure showing a fourth example of a region representation method in which the table of FIG. 11 according to this embodiment is expanded to three-dimensional elements. 11 is a diagram illustrating the configuration of a non-conformity countermeasure unit according to the present embodiment; FIG. 12 is a flowchart illustrating specific processing of a non-conformity countermeasure process according to the present embodiment; FIG. 13 is a diagram illustrating a configuration of a modified example of the non-conformity countermeasure unit of FIG. 15 according to the present embodiment; FIG. 14 is a diagram illustrating the hardware configuration of an image conversion system according to the present embodiment; FIG. 15 is an explanatory diagram illustrating how numerical values ​​in a table of model adaptation regions, which are expressed in the same manner as in FIG. 11 according to the present embodiment, change due to re-learning; FIG. 16 is a graph illustrating the occurrence of errors in low-quality images before processing by the image conversion unit (before image quality improvement) according to the present embodiment; FIG. 17 is a graph illustrating the occurrence of errors in equivalent-to-high-quality images after processing by the image conversion unit (after image quality improvement) according to the present embodiment; FIG. 18 is a diagram illustrating the state of measurement errors when images are converted using a learning model in which the deviation amount of paired learning images according to the present embodiment is a model adaptation region [-5, 5). FIG. 19 is a graph illustrating the occurrence of errors based on the results of the first learning round according to the present embodiment.10 is a graph showing the occurrence of errors based on the results of the second learning round according to the present embodiment.

[0011] An image conversion system for converting a low-quality image into a high-quality image according to this embodiment will be described with reference to the drawings.

[0012] FIG. 1 is an external view showing the process of photographing a semiconductor circuit pattern formed in two upper and lower layers. In the photographing environment 31A, the subject of photographing is a semiconductor wafer in which an upper layer circuit 101 and a lower layer circuit 102 are integrated into a multilayer structure (here, two upper and lower layers) formed by etching, impurity addition, and thin film formation. The electron microscope 31 photographs the semiconductor wafer by irradiating it with an electron beam (indicated by an arrow in the figure) from above (from the side closer to the upper layer circuit 101). The semiconductor wafer in the photographing environment 31A is a normal product in which there is no misalignment between the upper layer circuit 101 and the lower layer circuit 102.

[0013] In the imaging environment 31B, the object of imaging is a semiconductor wafer in which an upper layer circuit 101 and a lower layer circuit 102 are formed and integrated. The upper layer circuit 101 in the imaging environment 31B is formed in a state where it is displaced slightly to the left with respect to the lower layer circuit 102. Therefore, in the imaging environment 31B, it is necessary to detect that the semiconductor wafer is a defective product from the image captured by the electron microscope 31. In the imaging environment 31C as well, the upper layer circuit 101 is formed in a state where it is displaced slightly to the right with respect to the lower layer circuit 102. Therefore, in the imaging environment 31C as well, it is necessary to detect that the semiconductor wafer is a defective product from the image captured by the electron microscope 31.

[0014] 2 is an explanatory diagram of the circuit patterns formed on the semiconductor wafer of FIG. 1. Circuit patterns are formed on the upper layer circuit 101 and the lower layer circuit 102 by using a mask for each layer. To make the explanation easier to understand, the circuit pattern 101p of the upper layer circuit 101 is made slightly longer in the vertical direction than the circuit pattern 102p of the lower layer circuit 102. In reality, there are many circuit patterns formed on a single semiconductor wafer, but FIG. 2 intentionally shows a small number of circuit patterns for the sake of explanation.

[0015] FIG. 3 is an explanatory diagram showing the amount of deviation determined from the semiconductor wafer of FIG. 2. Captured images 111 to 114 show portions of captured images of a semiconductor wafer in which an upper layer circuit 101 and a lower layer circuit 102 are formed and integrated (FIG. 3 is a rough diagram, showing an enlarged portion of the circuit patterns after superposition). As shown in FIG. 1, the electron beam of the electron microscope 31 passes through the upper layer circuit 101 on the front side and the lower layer circuit 102 on the back side, so both circuit patterns are captured in the captured images. For example, in each of the captured images 111 to 114, a first circuit pattern (one of the circuit patterns 102p), a second circuit pattern 101p (one of the circuit patterns 101p), and a third circuit pattern 102p (one of the circuit patterns 102p) are captured side by side in the left-right direction, in that order from left to right.

[0016] The captured images 111 and 112 are images of a normal product with no misalignment, in which the first, second, and third circuit patterns are spaced apart at equal horizontal distances d. In other words, the amount of misalignment when this distance d is used as a reference is zero. The captured image 111 is a low-quality image that contains not only the circuit pattern but also coarse resolution, white noise, distortion, etc. (In the illustration, the white noise is represented by hatching.) The captured image 112 is a high-quality image with high resolution and little noise or distortion. It has the same circuit pattern arrangement as the captured image 111, but does not contain white noise. Here, a pair of learning images is a pair of images that serve as learning materials for training the learning model 14, with one of the pair serving as input data for the learning model 14 and the other serving as output data for the learning model 14. For example, the learning model 14, which performs image conversion processing for noise, inputs a low-quality image such as the captured image 111 and outputs a high-quality image such as the captured image 112. At this time, a pair of photographed images 111 and 112 showing the same subject is set as a pair of learning images.

[0017] Hereinafter, when the trained learning model 14 is operated, the learning model 14 receives the captured image 113 before image conversion as input data and the captured image 114 after image conversion as output data. The captured image 113 is a low-quality image captured of a defective product with misalignment. In addition to unnecessary white noise, the distance d+10 between the second and third circuit patterns is greater than the distance d in the captured image 111 (the second circuit pattern is shifted to the left by a shift amount = +10). The captured image 114 is a high-quality image whose image quality has been improved by applying the learning model 14 to the captured image 113. As a result of the image quality improvement, the unnecessary white noise contained in the captured image 113 has been cleanly removed. However, as a side effect of the image quality improvement, the position of the circuit pattern in the captured image 113 has been altered so that the positional relationship of the circuit pattern in the captured image 114 is closer to that of the circuit pattern in the captured image 112 (the second circuit pattern is shifted to the left by a shift amount = +3).

[0018] In other words, as shown in the paired learning images, it is expected that the amount of deviation will not change before and after image conversion (no error will occur). However, an error of 10 - 3 = 7 occurs between the amount of deviation = +10 in captured image 113 and the amount of deviation = +3 in captured image 114. As a result, the following erroneous judgment occurs in the image judgment results due to the influence of the error, as shown below. - Normally, the photographed semiconductor wafer would be correctly judged to be defective based on the large amount of deviation of "+10" (a deviation greater than or equal to the threshold value of 5) in captured image 113. - However, the photographed semiconductor wafer is erroneously recognized as a normal product based on the small amount of deviation of "+3" (a deviation less than the threshold value of 5) in captured image 114.

[0019] As explained above, if the learning model 14 is created using a captured image 112 with a small amount of deviation, the layout information of the circuit pattern will also be learned in addition to image noise removal. Therefore, when a captured image 113 with a large amount of deviation is input, it is thought that processing is performed to bring the layout information of the captured image 112 learned during learning, resulting in the output of the captured image 114. Image noise removal is an effective image transformation for improving measurement accuracy, but such movement of the circuit pattern is an inappropriate image transformation that reduces measurement accuracy. A learning model that performs such inappropriate image transformation is called a "learning model incompatibility."

[0020] A learning model mismatch cannot be detected by visual inspection of the captured image 114. Even though image noise removal has been performed correctly, it is impossible to determine whether the circuit pattern movement is due to a natural movement (misalignment) or whether it is due to a learning model mismatch. Therefore, a mechanism for detecting learning model mismatch and issuing a notification of the mismatch is required. In other words, the image quality of the captured image 114 itself has been improved by removing white noise, making it difficult for an inspector to visually detect a misalignment error (change in the position of the circuit pattern) contained in the captured image 114. Therefore, the misalignment detection unit 10 of this embodiment, described in Figure 4 and subsequent figures, detects the misalignment error as a learning model mismatch and notifies the inspector of the detection result, thereby enabling the inspector to identify problems that would not be noticeable by visual inspection of the captured image.

[0021] FIG. 4 is a configuration diagram of an image conversion system that converts low-quality images into high-quality images. The image conversion system includes a non-conformity detection unit 10, a non-conformity countermeasure unit 20, an imaging device 30, an image utilization unit 40, and a control display unit 50. The non-conformity detection unit 10 detects non-conformity of a learning model 14 in machine learning, which is used for image conversion processing such as image quality improvement processing. The image conversion unit 12 of the non-conformity detection unit 10 uses the learning model 14 to convert a low-quality image 11 into a high-quality image. The converted high-quality image is used for image observation and image measurement. Generally, the following shooting conditions are used to capture high-quality images: - Increase the shooting time. - Capture multiple short-exposure images and calculate their cumulative average. - Irradiate strong illumination light.

[0022] However, there are also imaging devices 30 that cannot capture images under such imaging conditions. Examples of imaging devices 30 include an electron microscope 31 and an X-ray tomography device 32. The electron microscope 31 irradiates an object to be observed (e.g., a semiconductor wafer) with an electron beam to observe the state of a circuit pattern formed on the wafer. The electron beam irradiation can damage the circuit pattern, causing it to become thinner (shrink). Shrinkage is caused by long exposure times (including capturing multiple short-exposure images) and high acceleration voltages. Therefore, it is not possible to capture high-quality images frequently.

[0023] The X-ray tomography device 32 also suffers from the same problems as the electron microscope 31. The X-ray tomography device 32 irradiates the human body with X-rays to capture images. High-quality images can be captured by prolonged exposure or by increasing the X-ray intensity. However, this approach increases the X-ray exposure dose, making it difficult to capture high-quality images. Therefore, to minimize damage to the subject, it is better to capture low-quality images 11. The low-quality images 11 captured by the imaging device 30 in this manner are stored in the captured image storage unit 33. The image conversion unit 12 then converts the low-quality images 11 in the captured image storage unit 33 into equivalent-high-quality images 13 (images with image quality equivalent to high-quality images). While the configuration in which the low-quality images 11 are input to the incompatibility detection unit 10 via the captured image storage unit 33 has been described, they may also be input directly from the imaging device 30 to the incompatibility detection unit 10.

[0024] The high-quality equivalent image 13 output from the nonconformity detection unit 10 is input to the image utilization unit 40. The image utilization unit 40 includes an image observation processing unit 41, an image measurement processing unit 42, and an image classification processing unit 43. The image observation processing unit 41 performs various image processing, such as zooming, to observe the input image. The image measurement processing unit 42 measures the size of a shape using image processing. For example, the image measurement processing unit 42 performs image processing using the converted high-quality equivalent image 13 to extract edge portions in the circuit pattern 101p of the upper layer circuit 101 and the circuit pattern 102p of the lower layer circuit 102 shown in FIG. 2 . The image measurement processing unit 42 then extracts the distance d between the edges shown in FIG. 3 .

[0025] The image classification processing unit 43 processes the input image to determine what kind of object it will be classified into. The image utilization unit 40 also performs other image processing, not shown, such as image segmentation processing to classify image regions, in which processing performance is reduced with low-quality images 11. The control display unit 50 controls the image utilization unit 40 in various ways and displays the processing results.

[0026] FIG. 5 is a configuration diagram of the incongruity detection unit 10. The incongruity detection unit 10 includes an image conversion unit 12 and an incongruity detection unit 15. The incongruity detection unit 10 stores a low-quality image 11 (captured image 113 in FIG. 3), an equivalent high-quality image 13 (captured image 114 in FIG. 3), and a learning model 14. The image conversion unit 12 uses the learning model 14 to convert the input low-quality image 11 captured by the imaging device 30 into an equivalent high-quality image 13. The learning model 14 is a machine-learned model such as a CNN (Convolutional Neural Network). The CNN is a means for converting the low-quality image 11 into the equivalent high-quality image 13 using the learning model 14.

[0027] The incompatibility detection unit 15 detects whether the low-quality image 11 input as a processing target of the image conversion unit 12 is incompatible with the learning model 14. Information for the incompatibility detection unit 15 to detect incompatibility includes information on the model suitability region indicating the evaluation value (e.g., deviation amount) of the paired learning images used in the learning process of the learning model 14, which is associated with each learning model 14 and registered in the storage unit of the incompatibility detection unit 10. The information on the model suitability region is expressed, for example, as a range of deviation amounts of the paired learning images used during learning. The incompatibility detection unit 15 determines that the learning model 14 is incompatible when the evaluation value of the input low-quality image 11 is not within the model suitability region. The incompatibility notification unit 16 notifies the inspector, as a detection result of the incompatibility detection unit 15, by means of a presentation means such as a screen display or audio, that the learning model 14 used in the conversion process of the low-quality image 11 processed by the image conversion unit 12 is incompatible.

[0028] FIG. 6 is a configuration diagram of the model learning unit 10B. The model learning unit 10B includes an image conversion unit 12 and a weight correction unit 12B. The nonconformity detection unit 10 stores a low-quality image 11 (captured image 111 in FIG. 3), a high-quality equivalent image 13 (captured image 112 in FIG. 3), a high-quality correct image 13B, and a learning model 14. The high-quality correct image 13B is a high-quality image that serves as a target (correct answer) for image quality improvement for the low-quality image 11. In the case of semiconductor wafers, a test wafer may be used as the high-quality correct image 13B, or in the case of X-rays, an X-ray imaging phantom simulating a human body may be used. The high-quality correct image 13B is a learning pair image paired with the low-quality image 11, and is an image at the same position and angle of view as the low-quality image 11. Note that the imaging conditions for the high-quality correct image 13B involve a higher dose of electron beam or X-ray irradiation than the imaging conditions for the low-quality image 11. Once the learning model 14 is generated, the high-quality correct image 13B is no longer necessary when converting other low-quality images 11.

[0029] The weight correction unit 12B corrects the weights of the learning model 14 so that the image quality of the equivalent high-quality image 13 approaches that of the high-quality correct image 13B. The weights of the learning model 14 are, for example, weight coefficients of a CNN network. Therefore, in the initial state, no weights are set for the learning model 14. Therefore, there is a deviation between the equivalent high-quality image 13 obtained by image conversion of the low-quality image 11 by the image conversion unit 12 and the high-quality correct image 13B. The weight correction unit 12B calculates a correction amount for correcting the weight coefficients of the learning model 14 to correct the deviation, and modifies the learning model 14. The model learning unit 10B stores the distribution of evaluation values ​​of the learning images used in the learning stage of the learning model 14 as a model fitting region in the memory unit, corresponding to the learning model 14.

[0030] The weight correction process by the weight correction unit 12B is repeated using a plurality of paired learning images and ends when the amount of weight correction becomes small. At the end of the weight correction process, the deviation between the equivalent high-quality image 13 and the high-quality correct image 13B is minimized. By using the learning model 14 at the end of the weight correction process, the image conversion unit 12 can generate the equivalent high-quality image 13, whose image quality is close to that of the high-quality correct image 13B, from the low-quality image 11.

[0031] 7 is a flowchart showing the processing flow of the incongruity detection unit 10. The image conversion unit 12 acquires the low-quality image 11 to be processed from the captured image storage unit 33 (multidimensional image collection data DB) (S11). Alternatively, the image conversion unit 12 may acquire the low-quality image 11 directly from the imaging device 30. The image conversion unit 12 uses the learning model 14 to acquire (image conversion) a high-quality equivalent image 13 from the acquired low-quality image 11 (S12). The image measurement processing unit 42 performs the image measurement processing described in FIG. 4 using the acquired low-quality image 11 to calculate the amount of misalignment between the upper and lower layers (S13).

[0032] The incompatibility detection unit 15 acquires information about the model compatibility area registered in the DB of the learning model 14 (S14). The incompatibility detection unit 15 determines whether the deviation amount of the low-quality image 11 calculated in S13 is within the model compatibility area of ​​the learning model 14 (S15), and considers the learning model 14 outside the model compatibility area to be an incompatibility of the learning model. If the result in S15 is Yes, the incompatibility detection unit 15 issues a notification of incompatibility of the learning model 14 in S15 via the incompatibility notification unit 16 (S16). Furthermore, the incompatibility countermeasure unit 20 may implement countermeasures against the incompatibility (S17). On the other hand, if the result in S15 is No, the deviation amount is output (S18).

[0033] An example of the learning model mismatch determination process (S15) will be described below. FIG. 8 is a diagram showing a method for defining a section of the model-matching region. Two types of definition methods are illustrated below. Section definition method 121 defines the section of the deviation amount of the paired learning images used during learning as the model-matching region. In the example of FIG. 8, the model-matching region is the union of a first section ([-5, 5]) with a deviation amount of -5 to +5 and a second section ([15, 20]) with a deviation amount of +15 to +20. In FIG. 8, the sections are represented by hatched bars.

[0034] The interval definition method 122 defines a model fitting region that takes into account the error between the deviation amounts described in FIG. 3. In order to increase the reliability of the model fitting region, it is desirable to take this error into account and shrink both ends of each interval (first interval and second interval) toward the inside of the interval. The standard deviation of the error, 3σ, may be used as the amount of shrinkage. 3σ is a 99.7% confidence interval. In other words, the probability that a value near the boundary of the model fitting region in interval definition method 121 is outside the model fitting region in interval definition method 122 is 0.3% (= 100 - 99.7). In this way, a model fitting region with increased reliability may be defined.

[0035] FIG. 9 illustrates an example of a method for setting a model-fitting region. The following describes a method for setting the sections of the model-fitting region. The horizontal axis of each graph 131 and 132 represents the deviation amount. Graph 131 indicates the deviation amount value of the high-quality correct image 13B from a certain pair of training images with a black dot, and the influence curve of that deviation amount is indicated by a convex curve. The function used to calculate the influence curve and the width of the base are determined through experiments, etc. There are multiple paired training images, and the distribution density for each deviation amount also varies. The vertical axis of graph 131 indicates a threshold value for determining the influence of the deviation amount of the paired training images. If the influence curve is equal to or greater than this threshold, it is determined to be a model-fitting region. In FIG. 9, the first section ([-35, -25]), the second section ([-20, +5]), and the third section ([+10, +25]) are each determined to be within the model-fitting region. Graph 132 shows the portion determined to be a model-fitting region. When reliability and the like are taken into consideration, the model fitting region of the graph 132 may be reduced using the method described in the section definition method 122 of FIG.

[0036] An example of a representation method (region representation method) as a data format for the model-suitable region will now be described. FIG. 10 is a table showing a first example of a region representation method. In this table, the value of the model-suitable region obtained in FIG. 9 is represented by the deviation amount between the start point and the end point. In this case, the learning model mismatch determination process of the mismatch detection unit 15 may be a process of determining whether the deviation amount of the low-quality image 11 falls within the model-suitable region (S15) for each item number (#).

[0037] FIG. 11 is a table showing a second example of a region representation method. In this table, for each model-fitting region identifier (left column of the table), a flag (right column) is set indicating whether the region is a model-fitting region (=1) or not (=0). The model-fitting region identifier is a single value, but is treated as an identifier that identifies each model-fitting region with a range. For example, the identifier "-10" in the table represents a model-fitting region of [-10, -5). The symbol [ includes the boundary, and the symbol ) does not. In this example, the quantization of the model-fitting region is set to 5, and the deviation of the starting point is used as its identifier. Furthermore, the model learning unit 10B can change the resolution of the model-fitting region by changing the quantization number.

[0038] When the table of FIG. 11 is used, the learning model mismatch determination process (S15) of the mismatch detection unit 15 determines which model match region (left column of the table) the deviation amount of the low-quality image 11 belongs to and references the flag in the right column of the table to which it belongs; if the flag is 1, it is a model match region, and if it is 0, it is not a model match region. For example, if the deviation amount of the low-quality image 11 is a value of 16, the mismatch detection unit 15 assigns the region identifier "15" to the image because it is included in the model match region [15, 20]. The mismatch detection unit 15 then references the region identifier "15" → flag "0" in the table of FIG. 11 to determine that the image is model mismatched.

[0039] The representation method shown in Figure 11 can be easily made multidimensional. Up until now, only horizontal deviations have been considered as the target of the model adaptation region. In reality, vertical deviations also occur, and changes in image quality due to factors completely different from deviations, such as differences in the accelerating voltage of the electron microscope 31, can also occur. Learning using paired learning images is required for each element.

[0040] FIG. 12 is a table showing a third example of a region representation method in which the table of FIG. 11 is expanded to two-dimensional elements. While the identifier range in the table of FIG. 11 is -10 to 40, the identifier range in the table of FIG. 12 is -20 to 30. In the table of FIG. 12, the horizontal items indicate identifiers of model-suitable regions representing the horizontal deviation amount, and the vertical items indicate identifiers of model-suitable regions representing the vertical deviation amount. The numerical value inside the cell where the horizontal and vertical items intersect indicates a model-suitable region with a value of 1, and a non-model-suitable region with a value of 0. In this example, a horizontal deviation amount of [0, 20] indicates a model-suitable region, and a vertical deviation amount of [0, 10] indicates a model-suitable region. When the table of FIG. 12 is used, the learning model non-suitability determination process (S15) of the non-suitability detection unit 15 simply determines to which cell in the table the deviation amount of each element of the low-quality image 11 belongs and references the numerical value inside the cell to which it belongs.

[0041] Fig. 13 is a table showing a modification in which the names of directories in which paired learning images are stored are stored in the cell values ​​of the table in Fig. 12. If the character string inside the cell is not in a model-suitable region, it is represented by 0, and if it is in a model-suitable region, it is represented by a value other than 0.

[0042] FIG. 14 shows a data structure illustrating a fourth example of a region representation method in which the table of FIG. 11 is expanded to three-dimensional elements. The X-axis of FIG. 14 indicates the horizontal displacement, the Y-axis indicates the vertical displacement, and the Z-axis indicates the acceleration voltage. While representations of more than three dimensions cannot be illustrated, the region representation method can be expanded to n dimensions. When the data structure of FIG. 14 is used, the learning model incompatibility determination process (S15) of the incompatibility detection unit 15 determines which table cell the combination of the horizontal displacement, vertical displacement, and acceleration voltage of the low-quality image 11 belongs to and references the numerical value (not shown) within the corresponding cell. According to the first embodiment, it is possible to detect whether the learning model is incompatible with the image to be processed, and to issue a warning that the measurement result has a large error.

[0043] 15 is a configuration diagram of the inconformance countermeasure unit 20. The inconformance countermeasure unit 20 has a countermeasure method search unit 22, a countermeasure method presentation unit 23, a relearning data input unit 24, a usage model change unit 25, an existing model relearning unit 26, and a new model learning unit 27. The inconformance countermeasure unit 20 stores a countermeasure method DB 21. Details of the inconformance countermeasure unit 20 will be described below with reference to FIG. 16.

[0044] FIG. 16 is a flowchart showing the specific process of the incompatibility correction process (S17). This flowchart describes the correction process without operator intervention, followed by a modified example with operator intervention. The usage model change unit 25 searches for another model capable of processing the deviation (S171). The usage model change unit 25 determines whether the search in S171 was successful and whether another model was found (S172). If the result in S172 is Yes, the usage model change unit 25 changes the usage model from the current incompatibility model to the found another model (S173). The usage model is the learning model 14 used by the image conversion unit 12 for the conversion process. In other words, when the incompatibility detection unit 15 detects an incompatibility, the usage model change unit 25 searches the storage unit of the incompatibility correction unit 20 for another learning model 14 corresponding to a model-matching region that matches the evaluation value of the input low-quality image 11, and controls the image conversion unit 12 to use the other learning model 14 for the conversion process of the input low-quality image 11. This allows the image conversion unit 12 to perform appropriate (less error) image conversion processing based on the learning model 14 that is adapted to the low-quality image 11 to be processed, as changed in S173.

[0045] If the answer is No in S172, the countermeasure method search unit 22 acquires an existing learning model that can be retrained from the learning model 14 DB (S174). The model acquired in S174 may be the current incompatible model or another existing learning model. The countermeasure method search unit 22 determines whether or not the acquisition of the existing learning model in S174 was successful (S175). If the answer is Yes in S175, the existing model retraining unit 26 retrains the existing model (S176). In other words, when the incompatible detection unit 15 detects incompatibleness, the existing model retraining unit 26 retrains the incompatible learning model 14 based on additional high-quality correct images 13B, thereby expanding the model compatibility region of the learning model 14. If the answer is No in S175, the new model training unit 27 trains a new model (S177).

[0046] Therefore, the relearning data input unit 24 obtains high-quality images of teaching material data to be used in the relearning process (S176) of the existing model relearning unit 26 and the learning process (S177) of the new model learning unit 27. Note that an existing model is a model that has been learned one or more times, and its model adaptation region also includes one or more intervals. On the other hand, a new model is a model in an initial state that has not been learned, and its model adaptation region also does not include any intervals.

[0047] FIG. 17 is a configuration diagram showing a modified example of the nonconformity countermeasure unit 20 of FIG. 15 . In FIG. 17 , a relearning data collection unit 24b is provided instead of the relearning data input unit 24. The relearning data collection unit 24b acquires high-quality images without operator operation by performing automatic operation according to the programmed operating procedure of the imaging device 30. In other words, the relearning data collection unit 24b operates the imaging device 30 according to the pre-set operating procedure, thereby accepting input of additional high-quality supervised images 13B captured. The relearning data collection unit 24b may also perform other processes of FIG. 16 in the same manner as in the automatic operation.

[0048] Furthermore, the countermeasure method presenting unit 23 may present the following three countermeasure methods found by the countermeasure method searching unit 22 from the countermeasure method DB 21 and the work procedures required for those countermeasure methods to the user, and allow the user to select which countermeasure method to adopt. For this reason, the countermeasure method DB 21 stores information indicating what work should be done next when the nonconformity reporting unit 16 determines that the model is nonconformant. (Countermeasure method 1) Changing the usage model by the usage model changing unit 25 (S173). The countermeasure method presenting unit 23 may display candidates for the usage model to be changed, and the operator may confirm the candidate usage model that has been confirmed by pressing a confirmation button or the like.

[0049] (Countermeasure Method 2) The existing model is retrained by the existing model retraining unit 26 (S176, described in detail in Example 2). Note that retraining requires capturing high-quality images as training data. Capturing high-quality images requires a different procedure from normal imaging (capturing low-quality images 11). Therefore, the countermeasure method presenting unit 23 may display this procedure (recipe) on the screen in an easy-to-understand manner for the worker to support the operation. Therefore, the countermeasure method DB 21 stores, for example, a method for capturing additional high-quality correct images 13B for retraining, such as "Please shine a strong light on the subject to capture a high-quality image." The contents stored in this countermeasure method DB 21 are various messages to be displayed on the worker's screen. (Countermeasure Method 3) The new model training unit 27 trains a new model (S177). Similar to (Countermeasure Method 2), the countermeasure method presenting unit 23 may display the procedure (recipe) on the screen in an easy-to-understand manner for the worker.

[0050] FIG. 18 is a hardware configuration diagram of the image conversion system. Each processing unit of the image conversion system (the nonconformity detection unit 10, the nonconformity reporting unit 16, the nonconformity countermeasure unit 20, the image utilization unit 40, and the control display unit 50) is configured as a computer 900 having a CPU 901, a RAM 902, a ROM 903, a HDD 904, a communication I / F 905, an input / output I / F 906, and a media I / F 907. The HDD 904 is configured as a storage device that stores the learning model 14, for example. The communication I / F 905 is connected to an external communication device 915. The input / output I / F 906 is connected to an input / output device 916. The media I / F 907 reads and writes data from a recording medium 917. Furthermore, the CPU 901 controls each processing unit by executing a program (also called an application or an abbreviated version of the program) loaded into the RAM 902. This program can also be distributed via a communication line or recorded on a recording medium 917 such as a CD-ROM. Each processing unit of the image conversion system can be any hardware capable of performing image arithmetic processing. For example, it can be a device that performs arithmetic processing and is equipped with a processing unit such as a CPU or GPU, or a storage device such as an HDD, such as a computer, or a field-programmable gate array (FPGA) that can program logic arithmetic circuits, or it can be manufactured with dedicated hardware.

[0051] In Example 2, details of the re-learning process (S176) of the existing model by the existing model re-learning unit 26 will be described. As an example of the re-learning process, a method will be described in which a commonly used fine tuning adjustment process is performed on the learning model, and the model fitting region is expanded while adapting the learning model. In fine tuning, the existing model is used as the pre-learning state. The training pair images used for training are both the training pair images (already registered in the model fitting region) used when creating the existing model and the training pair images (before being registered in the model fitting region) from the region where the model fitting region is to be expanded.

[0052] By using both paired training images, a training model that matches both the model adaptation region covered by the initial training model and the model adaptation region to be added is generated. Since paired training images are also required for the extended region, a high-quality correct image 13B is required in addition to the low-quality image 11. Therefore, the high-quality correct image 13B must be captured. Fine-tuning enables model generation at a much higher speed than when an untrained training model is used as the initial state, and is effective in improving processing throughput. Fine-tuning is also suitable for creating a training model with multiple combined objectives, such as using a training model trained for noise removal as the initial state and training using paired training images for aberration improvement. In other words, the retrained training model can perform image conversion processing that improves both noise removal and aberration, for example.

[0053] Figure 19 is an explanatory diagram showing how the values ​​in the model adaptation region table, which is expressed in the same way as Figure 11, change during retraining. The table in Figure 19 combines the initial training model (columns 1 to 3), the model showing the results of the first retraining (columns 4 and 5), and the model showing the results of the second retraining (columns 6 and 7) into a single table. Because the model adaptation region identifiers are the same for all three models, only the first column is listed. Columns 2, 4, and 6 of the "Determination" column contain flags indicating whether the region is a model adaptation region (=1) or not (=0). Columns 3, 5, and 7 of the "Storage Destination" column indicate the directory name in which the training pair images used for fine-tuning and adjusting the training model are stored. The training pair images in this directory are used as training data during retraining.

[0054] The initial learning model indicates a model suitability region [-5, 5), the results of the first learning run are expanded to the model suitability region [-15, 5), and the results of the second learning run are expanded to the model suitability region [-40, 5). Furthermore, the paired learning images used in the first learning run are stored in the directory names "A003, A004", respectively. The paired learning images used in the second learning run are stored in the directory names "A005 to A009", respectively. For example, if the magnitude of the deviation of the newly processed low-quality image 11 is in the region [-15, -5), then in the initial learning model, some of the intervals [-15, -5) are outside the range and are non-suitable, but if the results of the first learning run are used, all of them will be suitable within the range.

[0055] Fig. 20 is a graph showing the occurrence of errors in the low-quality image 11 before processing by the image conversion unit 12 (before image quality improvement). Fig. 21 is a graph showing the occurrence of errors in the equivalent high-quality image 13 after processing by the image conversion unit 12 (after image quality improvement). The horizontal axis of the graph represents the amount of deviation calculated using the equivalent high-quality image 13. The vertical axis of the graph represents the error between the amount of deviation calculated using the low-quality image 11 or the equivalent high-quality image 13 and the amount of deviation calculated using the high-quality correct image 13B. The learning model 14 was created using paired learning images with upper and lower layer deviation amounts ranging from -40 to +5.

[0056] 200 images were measured to generate the graphs in Figures 20 and 21. Figure 21 shows that the error variance is smaller across all misalignment amounts compared to Figure 20. Smaller error variance indicates improved accuracy, demonstrating the improved measurement accuracy achieved by the image conversion unit 12. The horizontal axis represents the misalignment amount calculated using equivalent-high-quality images 13 rather than the misalignment amount calculated from high-quality images because, for the reasons described in the section on obtaining high-quality images, high-quality images cannot be used in the process of determining whether the learning model is incompatible. In Figure 21, the learning model was created using paired learning images with misalignment amounts between -40 and +5 for the upper and lower layers. Misalignment between the upper and lower layers occurs due to some kind of manufacturing process issue, and it is difficult to prepare paired learning images with such a wide range of misalignment amounts in actual use.

[0057] FIG. 22 shows the measurement error when an image is converted using a learning model in which the deviation of the paired learning images is in the model-suitable region [-5, 5]. In the model-suitable region of FIG. 22, the error variance is approximately the same as in FIG. 21, demonstrating an improvement in measurement accuracy. However, for images that are shifted more negatively than the model-suitable region, the error increases with increasing distance from the model-suitable region. Thus, if the deviation of the low-quality image 11 is included in the model-suitable region, the error variance is small and there is little inappropriate movement of the circuit pattern. If the deviation of the low-quality image 11 is not included in the model-suitable region, there is a large deviation in the error and a large amount of inappropriate movement of the circuit pattern.

[0058] 23 is a graph showing the occurrence of errors based on the results of the first learning round. Fine tuning has expanded the model adaptability range to the interval [-15, 5), and it can be seen that the error (vertical axis) within this interval has become smaller. Based on the model adaptability range [-15, 5) of this learning model 14, the mismatch detection unit 15 determines subsequent model adaptability.

[0059] 24 is a graph showing the occurrence of errors based on the results of the second learning. As a result of fine tuning, the model adaptability range has been expanded to the interval [-40, 5), and it can be seen that the error (vertical axis) within almost the entire interval has become smaller. Based on the model adaptability range [-40, 5) of this learning model 14, the mismatch detection unit 15 judges the subsequent model adaptability.

[0060] As shown in FIG. 9, the deviations of the acquired training pair images may be concentrated in one area or may become coarse. If many images are concentrated in a certain model-fitting region identifier area, the weight of that area may become large, resulting in unbalanced re-training. Therefore, it is desirable to keep the quantization width of the model-fitting region in a table representing the model-fitting region, such as that shown in FIG. 11, constant. This makes it possible to prevent such imbalances by keeping the number of training pair images in each model-fitting region constant.

[0061] In the mismatch detection unit 10 of the present embodiment described above, the mismatch detection unit 15 determines whether the learning model 14 used to convert the low-quality image 11 to the equivalent high-quality image 13 is compatible with the low-quality image 11. The mismatch notification unit 16 notifies the user that the equivalent high-quality image 13 is not the intended high-quality image because the learning model 14 is incompatible with the low-quality image 11 of the input data, based on the determination by the mismatch detection unit 15. This allows the user to understand the model mismatch even if the equivalent high-quality image 13 has been improved, thereby preventing erroneous judgment of image content based on the equivalent high-quality image 13.

[0062] Furthermore, the incompatibility countermeasure unit 20 takes countermeasures such as adding to the training data a training target image corresponding to the low-quality image 11 of the input data for which the training model 14 is determined to be incompatible, and performing re-training. This creates a training model 14 that is compatible with the low-quality image 11. By re-training the training model 14 in this way, it is possible to gradually expand the model compatibility region corresponding to the training model 14. For example, if the training model is compatible with noise but not with aberration, the existing model re-training unit 26 can generate a training model 14 that is compatible with both the noise image and the aberration image by re-training using both the noise image and the aberration image.

[0063] The present invention is not limited to the above-described embodiments and includes various modifications. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those including all of the described configurations. Furthermore, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, or to add the configuration of another embodiment to the configuration of one embodiment. Furthermore, it is possible to add, delete, or replace part of the configuration of each embodiment with other configurations. Furthermore, the above-described configurations, functions, processing units, processing means, etc. may be implemented in hardware, in part or in whole, by designing them as integrated circuits, for example. Furthermore, the above-described configurations, functions, etc. may be implemented in software, in which a processor interprets and executes a program that implements each function.

[0064] Information such as programs, tables, and files that realize each function can be stored in a memory, a recording device such as a hard disk or a solid-state drive (SSD), or a recording medium such as an integrated circuit (IC) card, an SD card, or a digital versatile disc (DVD). Cloud computing can also be utilized. Furthermore, control lines and information lines are shown as necessary for explanation, and not all control lines and information lines in the product are necessarily shown. In reality, it can be assumed that almost all components are interconnected. Furthermore, the communication means connecting each device is not limited to a wireless LAN, but may be changed to a wired LAN or other communication means.

[0065] DESCRIPTION OF SYMBOLS 10 Nonconformity detection unit (nonconformity detection device) 10B Model learning unit 11 Low-quality image (image before conversion) 12 Image conversion unit 12B Weight correction unit 13 High-quality equivalent image (image after conversion) 13B High-quality correct image (image for learning) 14 Learning model 15 Nonconformity detection unit 16 Nonconformity reporting unit 20 Nonconformity countermeasure unit (nonconformity detection device) 21 Countermeasure method DB 22 Countermeasure method search unit 23 Countermeasure method presentation unit 24 Re-learning data input unit 24b Re-learning data collection unit 25 Used model change unit 26 Existing model re-learning unit 27 New model learning unit 30 Imaging device 31 Electron microscope 32 X-ray tomography device 33 Captured image storage unit 40 Image utilization unit 41 Image observation processing unit 42 Image measurement processing unit 43 Image classification processing unit 50 Control display unit

Claims

1. A learning model for inputting or outputting an image and performing image conversion; a mismatch detection unit for detecting whether the learning model is mismatched; a non-conformity reporting unit that reports a detected non-conformity; a storage unit that stores a distribution of evaluation values ​​of the learning images used in a learning stage of the learning model as a model fitting region in association with the learning model, The non-conformity detection unit determines that the learning model is non-conformant when the evaluation value of the image is not within the range of the model conformity region. Non-conformity detection device.

2. The non-conformity detection device further includes a usage model changing unit, The use model change unit is characterized in that, when the incompatibility detection unit detects an incompatibility, the use model change unit searches the storage unit for another learning model corresponding to the model compatibility area that matches the evaluation value of the image, and uses the other learning model in the conversion process of the image. The non-conformity detection device according to claim 1 .

3. The non-conformity detection device further includes an existing model re-learning unit, The existing model re-learning unit is characterized in that, when the incompatibility detection unit detects an incompatibility, the existing model re-learning unit re-learns the incompatibility learning model based on additional learning images, thereby expanding the model compatibility region of the learning model. The non-conformity detection device according to claim 1 .

4. The non-conformity detection device further includes a countermeasure method presentation unit and a re-learning data input unit, the countermeasure method presentation unit presents a countermeasure method for the non-conformity including a method for photographing the additional learning image; The re-learning data input unit receives input of additional learning images captured by a presented imaging method. The non-conformity detection device according to claim 3.

5. The non-conformity detection device further includes a re-learning data collection unit, The re-learning data collection unit operates an imaging device according to a preset operation procedure, and accepts input of the additional learning images captured. The non-conformity detection device according to claim 3.

6. The non-conformity detection device is A learning model for inputting or outputting an image and performing image conversion; a mismatch detection unit for detecting whether the learning model is mismatched; a non-conformity reporting unit that reports a detected non-conformity; a storage unit that stores a distribution of evaluation values ​​of the learning images used in a learning stage of the learning model as a model fitting region in association with the learning model, The non-conformity detection unit determines that the learning model is non-conformant when the evaluation value of the image is not within the range of the model conformity region. Non-conformance detection methods.

7. The image is an image of a circuit pattern formed on a semiconductor wafer, The evaluation value of the image is the amount of deviation, aberration, or noise removal of the circuit pattern obtained from the image. The non-conformity detection device according to claim 1 .