Unsuitability detection device and unsuitability detection method

Abstract

The unsuitability detection unit (10) is provided with: an image conversion unit (12) that uses a learning model (14) to convert an input low-quality image (11) into a high-quality equivalent image (13); an incompatibility detection unit (15) that detects whether or not the input low-quality image (11) is incompatible with the learning model (14); an unsuitability reporting unit (16) that reports the detected unsuitability; and a storage unit that stores the distribution of the evaluation values of the high-quality positive solution image (13B) used in the learning stage of the learning model (14) as a model-suited region in association with the learning model (14), and the unsuited detection unit (15) determines that the learning model (14) is unsuited when the evaluation value of the input low-quality image (11) is not within the range of the model-suited region.

Description

Technical Field

[0001] The present invention relates to an unsuitability detection device and an unsuitability detection method. Background Art

[0002] Patent document 1 describes a system for transforming a low-quality image into a high-quality image using machine learning. In the system described in Patent document 1, a learning model for machine learning is generated using a low-quality image and a high-quality image. Furthermore, in the system described in Patent document 1, the low-quality input image is transformed into a high-quality output image through machine learning using the learning model.

[0003] Prior art literature

[0004] Patent Literature

[0005] Patent Document 1: International Publication No. 2021 / 095256 Summary of the invention

[0006] Problems to be solved by the invention

[0007] In the system described in Patent Document 1, a learning model is prepared in advance according to the purpose of image quality improvement. For example, if the purpose is noise removal, a learning model corresponding to the size of the noise is prepared. Alternatively, if the purpose is aberration improvement, a learning model corresponding to the size of the aberration is prepared.

[0008] The user selects the learning model to be used based on the purpose of improving the image quality and the state of the image quality of the low-quality image (the state of noise and aberration). In addition, in the system described in Patent Document 1, the user visually selects the learning model for noise removal, so it is easy to determine whether the selected learning model is effective for noise removal.

[0009] In addition, the image transformation processing using the learning model has the property of transforming the input image in a manner close to the image of the teaching material data used when learning the learning model. Therefore, if a noisy input image is input to a learning model learned from teaching material data without noise, it can be expected that an output image with noise removed from the input image can be obtained in a manner close to the teaching material data.

[0010] On the other hand, sometimes image transformation processing with unexpected side effects that are different from the purpose of the learning model is also produced. For example, when the shape and position of subject A shown in the teaching material data are greatly different from the shape and position of subject B shown in the input image, subject B in the output image is deformed to be close to subject A. Due to such unexpected deformation, the meaning of the image changes, for example, it also affects the use scenario of the transformed image, such as visually checking whether the subject is a normal product or a defective product. The reason for this influence is that the learning model originally used for image transformation processing is not suitable for the input image.

[0011] Therefore, a main subject of the present invention is to detect the inappropriateness of the learning model used for image transformation processing.

[0012] Means for solving problems

[0013] In order to solve the above-mentioned problems, the unsuitability detection device of the present invention has the following features.

[0014] The present invention is characterized in that it comprises: an image transformation unit, which uses a learning model to transform an input pre-transformation image into a post-transformation image; an unsuitability detection unit, which detects whether the pre-transformation image is unsuitable for the learning model; an unsuitability reporting unit, which reports the detected unsuitability; and a storage unit, which stores the distribution of evaluation values ​​of the learning images used in the learning stage of the learning model as a model suitability area in correspondence with the learning model, and the unsuitability detection unit determines that the learning model is unsuitable when the evaluation value of the pre-transformation image is not within the range of the model suitability area.

[0015] Other means will be described later.

[0016] Effects of the Invention

[0017] According to the present invention, it is possible to detect inappropriateness of a learning model used in image conversion processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 This is an external view showing a circuit pattern of a semiconductor formed in upper and lower layers according to the present embodiment.

[0019] Figure 2 This embodiment is formed in Figure 1 An illustration of a circuit pattern of a semiconductor wafer.

[0020] Figure 3 This embodiment shows Figure 2 An illustration of the offset amount obtained for a semiconductor wafer.

[0021] Figure 4This is a diagram showing the configuration of an unsuitability detection unit used in the image conversion system for converting a low-quality image into a high-quality image according to the present embodiment.

[0022] Figure 5 It is a structural diagram of the unsuitability detection unit according to this embodiment.

[0023] Figure 6 This is a configuration diagram of the model learning unit of this embodiment.

[0024] Figure 7 This is a flowchart showing the processing flow of the unsuitability detection unit according to the present embodiment.

[0025] Figure 8 2 is a diagram showing a method of defining a section of a model fit region according to the present embodiment. Two definition methods are shown below.

[0026] Fig. 9 This is a diagram showing an example of a method for setting a model fit region according to the present embodiment.

[0027] Fig.10 This is a table showing a first example of a region expression method according to this embodiment.

[0028] Fig.11 This is a table showing a second example of the area expression method according to this embodiment.

[0029] Fig.12 This means that the Fig.11 A table of a third example of a method of expressing an area obtained by expanding a table of into two-dimensional elements.

[0030] Fig.13 In this embodiment, Fig.12 A table storing a variation of the directory name of the learning pair images is stored in the cell values ​​of the table.

[0031] Fig.14 This means that the Fig.11 The data structure of the fourth example of the method of expressing the area obtained by expanding the table into three-dimensional elements.

[0032] Fig.15 It is a structural diagram of the incompatibility countermeasure unit of this embodiment.

[0033] Fig.16 This is a flowchart showing the specific process of the inappropriate countermeasure process according to the present embodiment.

[0034] Fig.17 This embodiment shows Fig.15 A structural diagram of a modified example of the unsuitability countermeasure unit.

[0035] Fig.18This is a hardware configuration diagram of the image conversion system according to this embodiment.

[0036] Fig.19 This means that Fig.11 A table of model fit areas with the same representation method, illustrating how the values ​​in the table change due to relearning.

[0037] Fig. 20 This is a graph showing the occurrence of errors in a low-quality image before processing (before image quality improvement) by the image conversion unit of the present embodiment.

[0038] Fig.21 This is a graph showing the state of error occurrence in a high-quality equivalent image after processing (after image quality improvement) by the image conversion unit of the present embodiment.

[0039] Fig. 22 This is a diagram showing the state of measurement error when image transformation is performed using the learning model in which the offset amount of the learning paired image according to the present embodiment is set to the model fit region [-5, 5).

[0040] Fig.23 This is a graph showing the error occurrence status based on the result of the first learning according to the present embodiment.

[0041] Fig.24 This is a graph showing the error occurrence status based on the result of the second learning according to the present embodiment. DETAILED DESCRIPTION

[0042] An image conversion system for converting a low-quality image into a high-quality image according to the present embodiment will be described with reference to the drawings.

[0043] Example 1

[0044] Figure 1 This is an external view showing a photograph of a circuit pattern of a semiconductor formed on upper and lower layers.

[0045] In the shooting environment 31A, the semiconductor wafer in which the upper circuit 101 and the lower circuit 102 are formed into a multilayer structure (here, two layers) and integrated by etching, impurity addition, and thin film formation is the shooting object. The electron microscope 31 irradiates the electron beam (arrow in the figure) from the upper part (from the side close to the upper circuit 101) to shoot the semiconductor wafer. The semiconductor wafer in the shooting environment 31A is a normal product with no positional offset between the upper circuit 101 and the lower circuit 102.

[0046] In the photographing environment 31B, a semiconductor wafer formed by forming an upper circuit 101 and a lower circuit 102 is an integrated semiconductor wafer. The upper circuit 101 of the photographing environment 31B is formed in a state slightly offset to the left relative to the lower circuit 102. Therefore, in the photographing environment 31B, it is necessary to detect whether the semiconductor wafer is a defective product based on the image taken by the electron microscope 31.

[0047] In the imaging environment 31C, the upper circuit 101 is also formed in a state slightly shifted to the right with respect to the lower circuit 102. Therefore, in the imaging environment 31C, it is necessary to detect whether the semiconductor wafer is defective based on the image captured by the electron microscope 31.

[0048] Figure 2 is formed in Figure 1 An illustration of a circuit pattern of a semiconductor wafer.

[0049] Circuit patterns are formed by using masks for each layer in the upper layer circuit 101 and the lower layer circuit 102. For easier understanding of the description, the circuit pattern 101p of the upper layer circuit 101 is slightly longer than the circuit pattern 102p of the lower layer circuit 102 in the vertical direction.

[0050] In fact, there are multiple circuit patterns formed on one semiconductor wafer, but Figure 2 A few circuit patterns are shown in the figure for illustration purpose.

[0051] Figure 3 It means from Figure 2 An illustration of the offset amount obtained for a semiconductor wafer.

[0052] The captured images 111 to 114 show a portion of the captured images of the semiconductor wafer on which the upper layer circuit 101 and the lower layer circuit 102 are formed and integrated ( Figure 3 This is a rough diagram, and a part of the overlapping circuit pattern is extracted and enlarged. Figure 1 As shown, the electron beam of the electron microscope 31 passes through the upper layer circuit 101 on the front side and the lower layer circuit 102 on the back side, so both circuit patterns appear in the captured image.

[0053] For example, in captured images 111 to 114, the first circuit pattern (one of the circuit patterns 102p), the second circuit pattern 101p (one of the circuit patterns 101p), and the third circuit pattern 102p (one of the circuit patterns 102p) are respectively arranged in the left-right direction from left to right.

[0054] The captured images 111 and 112 are captured images of normal products without positional displacement, and the first, second, and third circuit patterns are arranged at an equal distance d in the left-right direction. That is, the displacement amount based on the distance d is 0.

[0055] The captured image 111 is a low-quality image containing, in addition to the circuit pattern, white noise (noise), distortion, and the like with coarse resolution (in the figure, the white noise is expressed by hatching).

[0056] The captured image 112 is a high-quality image with high resolution and little noise and distortion. Although the arrangement of the circuit pattern is the same as that of the captured image 111 , white noise is not captured.

[0057] Here, the paired images for learning refer to a pair of images that serve as teaching materials when learning the learning model 14, one of the paired images serves as input data to the learning model 14, and the other of the paired images serves as output data to the learning model 14. For example, when a low-quality image such as the captured image 111 is input to the learning model 14 that performs image conversion processing such as noise, a high-quality image such as the captured image 112 is output. In this case, a pair of the captured image 111 and the captured image 112 showing the same subject is set as the paired images for learning.

[0058] Next, when the learned learning model 14 is used, the learning model 14 receives the captured image 113 before the image conversion as input data and outputs the captured image 114 after the image conversion as output data.

[0059] Captured image 113 is a low-quality image of a defective product with positional displacement. In addition to unnecessary white noise, the distance d+10 between the second and third circuit patterns is larger than the distance d of captured image 111 (the second circuit pattern is offset to the left by an amount = +10).

[0060] The captured image 114 is a high-quality image whose image quality is improved by applying the learning model 14 to the captured image 113. As a function of the image quality improvement, the captured image 114 cleanly removes the unnecessary white noise contained in the captured image 113. However, as a side effect of the image quality improvement, the captured image 114 changes the position of the circuit pattern in the captured image 113 in a manner close to the positional relationship of the circuit pattern in the captured image 112 (the second circuit pattern is shifted to the left by the shift amount = +3).

[0061] That is, as shown in the paired image for learning, it is expected that the offset will not change (no error will occur) before and after the image transformation. However, an error of 10-3=7 occurs between the offset = +10 in the captured image 113 and the offset = +3 in the captured image 114. As a result, the image determination result also produces the following erroneous determination due to the influence of the error, as shown below.

[0062] Originally, based on a large offset of “+10” (threshold=offset of 5 or more) in the captured image 113 , the captured semiconductor wafer can be correctly determined as a defective product.

[0063] However, based on a small offset amount of “+3” in the captured image 114 (an offset amount smaller than the threshold value=5), the captured semiconductor wafer is mistakenly recognized as a normal product.

[0064] As described above, when learning is performed using the captured image 112 with a small amount of offset when preparing the learning model 14, in addition to removing noise from the image, the configuration information of the circuit pattern is also learned. Therefore, when the captured image 113 with a large amount of offset is input, processing is performed to approximate the configuration information of the captured image 112 learned during learning, and it is considered that the captured image 114 is output.

[0065] Image noise removal is an effective image transformation for improving measurement accuracy, but such movement of the circuit pattern is an inappropriate image transformation that reduces measurement accuracy. A learning model that performs such an inappropriate image transformation is defined as "learning model inappropriate".

[0066] Even if the captured image 114 is visually checked, it is not possible to determine that the learning model is not suitable, and normal image noise removal cannot determine whether the movement of the circuit pattern is natural (displacement occurs) or due to the learning model not being suitable. Therefore, a mechanism is needed to detect and report the learning model not being suitable.

[0067] That is, the captured image 114 itself is improved in image quality by removing white noise, and it is difficult for the inspector to visually notice the error in the offset (position change of the circuit pattern) included in the captured image 114. Figure 4 The incompatibility detection unit 10 of the present embodiment described later detects an error in the offset amount as incompatibility of the learning model 14 and notifies the inspector of the detection result, thereby allowing the inspector to grasp a problem that was not noticed by visual inspection of the captured image.

[0068] Figure 4 This is a structural diagram of an image conversion system that converts a low-quality image into a high-quality image.

[0069] The image conversion system includes an unsuitability detecting unit 10 , an unsuitability countermeasure unit 20 , an imaging device 30 , an image utilizing unit 40 , and a control and display unit 50 .

[0070] The unsuitable detection unit 10 detects unsuitability of a learning model 14 in machine learning used in image conversion processing such as image quality improvement processing. The image conversion unit 12 of the unsuitable detection unit 10 converts a low-quality image 11 into a high-quality image using the learning model 14. The converted high-quality image is used for image observation or image measurement. In general, the following shooting conditions are used to shoot a high-quality image.

[0071] Extend the shooting time.

[0072] Take multiple short-exposure images and calculate their cumulative average.

[0073] ·Illuminate with strong lighting.

[0074] However, there are also imaging devices 30 that cannot perform imaging under such imaging conditions. For example, the imaging device 30 includes an electron microscope 31 and an X-ray tomography imaging device 32. The electron microscope 31 irradiates an object (e.g., a semiconductor wafer) with an electron beam to observe the appearance of a circuit pattern formed on the wafer.

[0075] Sometimes, the circuit pattern is damaged by electron beam irradiation, causing the circuit pattern to become thinner (shrink). The shrinkage is caused by long exposure (including taking multiple short exposure images) and high acceleration voltage. Therefore, it is impossible to take high-quality images at a high frequency.

[0076] The same problem as that of the electron microscope 31 also occurs in the X-ray tomography apparatus 32. The X-ray tomography apparatus 32 irradiates the human body with X-rays to perform imaging. High-quality images can be captured by irradiating for a long time or increasing the X-ray intensity. However, this results in an increase in the amount of X-ray radiation, and therefore it is difficult to capture high-quality images by such a method. Therefore, in order to minimize damage to the subject, it is better to capture low-quality images 11.

[0077] The low-quality image 11 captured by the camera 30 in this manner is stored in the captured image storage unit 33. Then, the image conversion unit 12 converts the low-quality image 11 in the captured image storage unit 33 into a high-quality equivalent image 13 (an image having a quality equivalent to that of a high-quality image). In addition, although the structure in which the low-quality image 11 is input to the unsuitable detection unit 10 via the captured image storage unit 33 has been described, it may be directly input to the unsuitable detection unit 10 from the camera 30.

[0078] The high-quality equivalent image 13 output from the incompatibility detecting section 10 is input to the image utilizing section 40 .

[0079] The image utilization unit 40 includes an image observation processing unit 41 , an image measurement processing unit 42 , and an image classification processing unit 43 .

[0080] The image observation processing unit 41 performs various image processing such as enlargement and reduction in order to observe the input image.

[0081] The image measurement processing unit 42 uses image processing to measure the size of the shape. For example, the image measurement processing unit 42 uses the transformed high-quality equivalent image 13 to perform image processing. Figure 2 The edge portions are extracted from the circuit pattern 101p of the upper layer circuit 101 and the circuit pattern 102p of the lower layer circuit 102. Figure 3 The distance d between the edges is shown.

[0082] The image classification processing unit 43 processes what kind of object the input image is classified as. Although not shown, the image utilization unit 40 performs image processing in a processing area where processing performance is reduced in the low-quality image 11, such as image segmentation processing to classify image regions.

[0083] The control and display unit 50 performs various controls of the image utilization unit 40 and displays processing results.

[0084] Figure 5 2 is a diagram showing the structure of the unsuitable detection unit 10 .

[0085] The unsuitable detection unit 10 includes an image conversion unit 12 and an unsuitable detection unit 15. The unsuitable detection unit 10 stores the low-quality image 11 (in Figure 3 The captured image 113), the high-quality equivalent image 13 (in Figure 3 , wherein the captured image 114) and the learning model 14.

[0086] The image conversion unit 12 outputs a high-quality equivalent image 13 using a learning model 14 based on the input low-quality image 11 captured by the camera 30. The learning model 14 is, for example, a model that has undergone machine learning such as CNN (Convolutional Neural Network). CNN is a means of converting a low-quality image 11 into a high-quality equivalent image 13 using the learning model 14.

[0087] The unsuitability detection unit 15 detects whether the low-quality image 11 input as the processing target of the image conversion unit 12 is unsuitable with the learning model 14. In addition, as information used by the unsuitability detection unit 15 to detect unsuitability, information on a model suitable area indicating an evaluation value (shift amount, etc.) of a learning paired image used in the learning process of the learning model 14 is registered in the storage unit of the unsuitability detection unit 10 in correspondence with each learning model 14. The information on the model suitable area is expressed as, for example, an interval of the shift amount of the learning paired image used in the learning process.

[0088] Then, when the evaluation value of the input low-quality image 11 is not within the range of the model applicable region, the unsuitability detecting unit 15 determines that the learning model 14 is unsuitable.

[0089] As a detection result of the unsuitability detecting unit 15 , the unsuitability notifying unit 16 notifies the examiner through a notification means such as a screen display or sound that the learning model 14 used in the conversion process of the low-quality image 11 to be processed by the image converter 12 is unsuitable.

[0090] Figure 6 This is a diagram showing the structure of the model learning unit 10B.

[0091] The model learning unit 10B includes an image conversion unit 12 and a weight correction unit 12B. The unsuitable detection unit 10 stores the low-quality image 11 ( Figure 3 The captured image 111), the high-quality equivalent image 13 ( Figure 3 : captured image 112), high-quality correct answer image 13B and learning model 14.

[0092] The high-quality correct image 13B is a high-quality image that is a target (correct answer) for image quality improvement of the low-quality image 11. The high-quality correct image 13B may be a test wafer in the case of a semiconductor wafer or an X-ray imaging phantom simulating a human body in the case of X-rays.

[0093] The high-quality correct answer image 13B is a paired image for learning that is paired with the low-quality image 11 , and is an image at the same position and angle of view as the low-quality image 11 .

[0094] Furthermore, the imaging condition of the high-quality correct image 13B has a higher irradiation amount of electron beams or X-rays than the imaging condition of the low-quality image 11. Once the learning model 14 is generated, the high-quality correct image 13B is not required when converting the other low-quality images 11.

[0095] The weight correction unit 12B corrects the weight of the learning model 14 so that the image quality of the high-quality equivalent image 13 is close to the high-quality correct image 13B. The weight of the learning model 14 is, for example, a weight coefficient of a CNN network. Therefore, in an initial state, the weight of the learning model 14 is in a state where nothing is set. Therefore, an offset occurs between the high-quality equivalent image 13 obtained by the image conversion unit 12 performing image conversion on the low-quality image 11 and the high-quality correct image 13B.

[0096] The weight correction unit 12B calculates a correction amount for correcting the weight coefficient of the learning model 14 for correcting the deviation amount, and corrects the learning model 14 .

[0097] The model learning unit 10B stores the distribution of evaluation values ​​of the learning images used in the learning stage of the learning model 14 as a model fit region in the storage unit in association with the learning model 14 .

[0098] The weight correction process of the weight correction unit 12B is repeatedly performed using a plurality of paired images for learning, and ends when the weight correction amount becomes small. At the end of the weight correction process, the deviation between the high-quality equivalent image 13 and the high-quality correct image 13B is minimized. The image conversion unit 12 can generate a high-quality equivalent image 13 with a quality close to that of the high-quality correct image 13B from the low-quality image 11 by using the learning model 14 at the end of the weight correction process.

[0099] Figure 7 : is a flowchart showing the processing flow of the unsuitability detection unit 10 .

[0100] The image converter 12 obtains the low-quality image 11 to be processed from the captured image storage unit 33 (multi-dimensional acquisition data DB for images) ( S11 ). Alternatively, the image converter 12 may directly obtain the low-quality image 11 from the imaging device 30 .

[0101] The image conversion unit 12 obtains a high-quality equivalent image 13 from the obtained low-quality image 11 using the learning model 14 (image conversion) ( S12 ).

[0102] The image measurement processing unit 42 uses the acquired low-quality image 11 to perform Figure 4 The image measurement process described above is performed to calculate the offset between the upper and lower layers (S13).

[0103] The unsuitability detecting unit 15 acquires information on the model suitable region registered in the DB of the learning model 14 ( S14 ).

[0104] The unsuitability detection unit 15 determines whether the offset of the low-quality image 11 calculated in S13 is within the model fit region of the learning model 14 ( S15 ), and regards the learning model 14 outside the model fit region as unsuitable.

[0105] If the answer is "Yes" in S15, the unsuitability detection unit 15 reports the unsuitability of the learning model 14 in S15 via the unsuitability reporting unit 16 (S16). In addition, the unsuitability countermeasure unit 20 may also execute unsuitability countermeasures (S17). On the other hand, if the answer is "No" in S15, the offset is output (S18).

[0106] An example of the learning model inappropriateness determination process ( S15 ) will be described below.

[0107] Figure 8This is a diagram showing a method of defining the interval of the model fit region. Two definition methods are illustrated below.

[0108] The interval definition method 121 directly defines the interval of the offset of the paired image for learning used in learning as the model fit region. Figure 8 In the example, the model fit area is the union of the first interval ([-5,5]) with an offset of -5 to +5 and the second interval ([15,20]) with an offset of +15 to +20. Figure 8 The shaded bars in the graph represent the range.

[0109] The interval definition method 122 defines the considerations Figure 3 The offsets in the model are the errors from each other in the fit region.

[0110] In order to improve the reliability of the model fit region, it is preferred to consider the error and reduce the two ends of the interval toward the inside of the interval for each interval (in the first interval and the second interval). As the amount of reduction, the standard deviation of the error, 3σ, etc., can also be used. 3σ is a 99.7% confidence interval. That is, the probability that the value near the boundary in the model fit region of the interval definition method 121 is outside the model fit region in the interval definition method 122 is 0.3% (=100-99.7). In this way, the definition of the model fit region with improved reliability can be performed.

[0111] Fig. 9 This is a diagram showing an example of a method of setting a model adaptation area.

[0112] The method of setting the interval of the model fit region will be described. The horizontal axis of each of the graphs 131 and 132 represents the offset.

[0113] In the graph 131, the black dots represent the offset values ​​of the high-quality correct image 13B in a certain paired image for learning, and the influence curve of the offset is represented by a mountain-shaped curve. The function used in the calculation of the influence curve, the width of the foot of the mountain, etc. are obtained by experiments, etc. There are multiple paired images for learning, and the distribution density of the offset is also different.

[0114] The vertical axis of the graph 131 records a threshold value for determining the influence of the offset amount of the paired image for learning. If the influence curve is greater than the threshold value, it is determined to be a model-fitting region. Fig. 9 In the data, the first interval ([-35, -25]), the second interval ([-20, +5]), and the third interval ([+10, +25]) are respectively determined to be intervals within the model suitable area.

[0115] The graph 132 shows the part determined as the model suitable area. In consideration of reliability, etc., the model suitable area based on the graph 132 is used in Figure 8The method described in the interval definition method 122 can be used to reduce the model suitable area.

[0116] An example of a method of expressing a data format suitable for a model region (region expression method) will be described.

[0117] Fig.10 This is a table showing a first example of a method of expressing an area.

[0118] In this table, the offset between the start point and the end point is used to represent Fig. 9 In this case, the learning model unsuitability determination process of the unsuitability detection unit 15 may perform a determination process (S15) for each item number (#) to determine whether the offset of the low-quality image 11 is within the model suitable region.

[0119] Fig.11 This is a table showing a second example of the area expression method.

[0120] In this table, for each model-fitting region identifier (left column of the table), a flag (right column) is created to indicate whether it is a model-fitting region ((=1) or (=0)).

[0121] The identifier of the model suitable area is 1 value, but it is treated as an identifier for identifying each model suitable area with a width. For example, the identifier "-10" recorded in the table indicates that the model suitable area is [-10, -5). The symbol [includes the boundary, the symbol) does not include the boundary. In this example, the quantization of the model suitable area is set to 5, and the offset of the starting point is used as its identifier. In addition, the model learning unit 10B can change the resolution of the model suitable area by changing the quantization number.

[0122] In use Fig.11 In the case of the table, the learning model unsuitability judgment processing (S15) of the unsuitability detection unit 15 determines to which model suitable area (left column of the table) the offset of the low-quality image 11 belongs, by referring to the flag in the right column of the table to which it belongs. If the flag is 1, it is a model suitable area, and if it is 0, it is not a model suitable area.

[0123] For example, when the offset of the low-quality image 11 is 16, since it is included in the model suitable area [15, 20), the unsuitable detection unit 15 sets the area identifier to "15". Then, the unsuitable detection unit 15 calculates the area identifier from Fig.11 In the table, the reference area identifier "15" → flag "0" is judged as the model is not suitable.

[0124] in addition, Fig.11The expression method of is easy to multi-dimensionalize. So far, only the horizontal offset is taken as the object of the model fitting area. In fact, the vertical offset also occurs, and the image quality changes completely different from the offset, such as the difference in the acceleration voltage of the electron microscope 31, etc., also occur. It is necessary to learn using the learning paired image for each element.

[0125] Fig.12 It means that Fig.11 The table is expanded to the table of the third example of the method of expressing the area of ​​two-dimensional elements. Fig.11 In the table, the identifier range is set to -10 to 40, but in Fig.12 In the table, the range of the identifier is set to -20 to 30.

[0126] exist Fig.12 In the table, the horizontal item shows the identifier of the model suitable area indicating the horizontal offset, and the vertical item shows the identifier of the model suitable area indicating the vertical offset. The value inside the cell where the horizontal item and the vertical item intersect is 1 when the model suitable area is 1, and 0 when it is not a model suitable area. In this example, the horizontal offset [0,20) represents the model suitable area, and the vertical offset [0,10) represents the model suitable area.

[0127] In use Fig.12 In the case of a table, the learning model unsuitability determination process (S15) of the unsuitability detection unit 15 determines to which unit in the table the offset of each element of the low-quality image 11 belongs, and refers to the value inside the unit to which it belongs.

[0128] Fig.13 It means in Fig.12 A table storing a variation of the directory name of the paired images for learning is stored in the cell value of the table. The character string inside the cell is represented by 0 when it is not a model-fitting region, and is represented by other than 0 when it is a model-fitting region.

[0129] Fig.14 It means that Fig.11 The table is expanded to the data structure of the fourth example of the area expression method of the three-dimensional element.

[0130] Fig.14 The X-axis represents the horizontal displacement, the Y-axis represents the vertical displacement, and the Z-axis represents the acceleration voltage. In addition, representations above three dimensions cannot be illustrated, but the regional representation method can be extended to n dimensions.

[0131] In use Fig.14In the case of a data structure, the learning model of the unsuitable detection unit 15 is unsuitable for determination processing (S15) to determine to which unit in the table the combination of the horizontal offset, the vertical offset and the acceleration voltage of the low-quality image 11 belongs, by referring to the numerical value inside the corresponding unit (not shown in the figure).

[0132] According to the first embodiment, it is possible to detect that the learning model is not suitable for the processing target image, and to report that the error of the measurement result is large.

[0133] Fig.15 This is a diagram showing the structure of the incompatibility countermeasure unit 20 .

[0134] The unsuitable countermeasure unit 20 includes a countermeasure method search unit 22, a countermeasure method prompt unit 23, a relearning data input unit 24, a used model change unit 25, an existing model relearning unit 26, and a new model learning unit 27. The unsuitable countermeasure unit 20 stores a countermeasure method DB 21. Fig.16 The details of the incompatibility countermeasure unit 20 will be described.

[0135] Fig.16 This is a flowchart showing the specific process of the unsuitable countermeasure process (S17).

[0136] In this flowchart, a countermeasure process not involving an operator is described, and a modified example involving an operator is described later.

[0137] The model changing unit 25 searches for other models that can handle the offset (S171). The model changing unit 25 determines whether the search in S171 is successful and whether other models are found (S172). If "yes" in S172, the model changing unit 25 changes the model to be used from the current unsuitable model to the other model found (S173). The model to be used refers to the learning model 14 used by the image transformation unit 12 in the transformation process.

[0138] That is, when the unsuitability detection unit 15 detects unsuitability, the model changing unit 25 searches for other learning models 14 corresponding to the model suitable area suitable for the evaluation value of the input low-quality image 11 from the storage unit of the unsuitability countermeasure unit 20, and controls the image transformation unit 12 to use the other learning models 14 for the transformation processing of the input low-quality image 11.

[0139] Thus, the image conversion unit 12 can perform appropriate (less error) image conversion processing based on the learning model 14 that is changed in S173 and is suitable for the low-quality image 11 to be processed.

[0140] If "No" in S172, the countermeasure search unit 22 obtains an existing learning model that can be relearned from the DB of the learning model 14 (S174). The model obtained in S174 may be an unsuitable model of the current situation or another existing learning model. The countermeasure search unit 22 determines whether the acquisition of the existing learning model in S174 is successful (S175).

[0141] If the answer in S175 is "yes", the existing model relearning unit 26 relearns the existing model (S176). That is, when the unsuitable detection unit 15 detects unsuitable, the existing model relearning unit 26 relearns the unsuitable learning model 14 based on the additional high-quality correct image 13B, thereby expanding the model suitable area of ​​the learning model 14.

[0142] If "NO" in S175, the new model learning unit 27 learns a new model (S177).

[0143] Therefore, the relearning data input unit 24 obtains a high-quality image of the teaching material data used in the relearning process (S176) of the existing model relearning unit 26 and the learning process (S177) of the new model learning unit 27. In addition, the existing model refers to a model that has been learned more than once, and its model suitable area also includes one or more intervals. On the other hand, the new model is a model in an initial state that has not been learned, and its model suitable area also does not include intervals.

[0144] Fig.17 Yes means Fig.15 A structural diagram of a modified example of the unsuitability countermeasure unit 20.

[0145] exist Fig.17 In the embodiment, a relearning data collection unit 24b is provided in place of the relearning data input unit 24. The relearning data collection unit 24b obtains a high-quality image without an operator's operation by executing an automatic operation according to a programmed operation procedure of the imaging device 30.

[0146] That is, the relearning data collection unit 24b operates the imaging device 30 according to a pre-set operation sequence, thereby accepting the input of the additional high-quality correct answer image 13B that has been captured. Fig.16 The other processing is also performed automatically.

[0147] In addition, the countermeasure method prompting unit 23 may also prompt the user with the following three countermeasure methods and the operation steps required for the countermeasure methods that the countermeasure method search unit 22 has searched from the countermeasure method DB 21, so that the user can select which countermeasure method to adopt. Therefore, information indicating what operation is appropriate to perform next when the inappropriateness reporting unit 16 determines that the model is inappropriate is registered in the countermeasure method DB 21.

[0148] (Countermeasure 1) Change of usage model by the usage model change unit 25 (S173) The countermeasure presentation unit 23 may display candidates of usage models to be changed, and identify the candidate of usage model confirmed by the operator pressing a confirmation button or the like.

[0149] (Countermeasure method 2) Relearning of the existing model by the existing model relearning unit 26 (S176, described in detail in Example 2). In addition, when relearning is performed, it is necessary to capture high-quality images as learning data. The capture of high-quality images must be performed in an operating procedure that is different from the usual capture (capturing of low-quality images 11). Therefore, the countermeasure method prompt unit 23 can also display the operating procedure (process) on the screen in a manner that is easy for the operator to understand, and provide support for the operation. Therefore, in the countermeasure method DB21, for example, methods for capturing additional high-quality correct images 13B for relearning are saved, such as "In order to capture high-quality images, please irradiate the subject with strong light." The storage content of the countermeasure method DB21 is various messages displayed on the operator's screen.

[0150] (Countermeasure 3) Learning of a new model by the new model learning unit 27 (S177) As in (Countermeasure 2), the countermeasure presentation unit 23 may display the operation steps (recipe) on the screen so that the operator can understand easily.

[0151] Fig.18 It is the hardware structure diagram of the image transformation system.

[0152] Each processing unit of the image conversion system (unsuitable detection unit 10, unsuitable notification unit 16, unsuitable countermeasure unit 20, image utilization unit 40, and control display unit 50) is configured as a computer 900 having a CPU 901, a RAM 902, a ROM 903, a HDD 904, a communication I / F 905, an input / output I / F 906, and a medium I / F 907. The HDD 904 is configured as a storage device that stores the learning model 14, for example.

[0153] The communication I / F 905 is connected to an external communication device 915. The input / output I / F 906 is connected to an input / output device 916. The medium I / F 907 reads and writes data from a recording medium 917. In addition, the CPU 901 controls each processing unit by executing a program (also referred to as an application program or its abbreviation application) read into the RAM 902. Moreover, the program can also be distributed via a communication line or recorded in a recording medium 917 such as a CD-ROM.

[0154] In addition, each processing unit of the image conversion system may be any hardware that can perform image processing. For example, it may be a device that carries out processing using a CPU, GPU or other processing device, or a storage device such as an HDD, as in a computer, or an FPGA (field-programmable gate array) that can program logic circuits, or it may be a dedicated hardware.

[0155] Example 2

[0156] In the second embodiment, the details of the relearning process (S176) of the existing model performed by the existing model relearning unit 26 are described. As an example of the relearning process, a method of adjusting the learning model using the generally used fine-tuning to make the learning model fit while expanding the model fit region is described. In the fine-tuning, the existing model is used as the state before learning. The learning paired images used in the learning are both the learning paired images (registered in the model fit region) used when the existing model was created and the learning paired images (registered before the model fit region) of the region that expands the model fit region.

[0157] By using the paired images for learning of both sides, a learning model suitable for both the model suitable area covered by the learning model in the initial state and the additional model suitable area is generated. In addition, the extended area also requires a paired image for learning, so a high-quality correct image 13B is required in addition to the low-quality image 11. Therefore, a shooting operation of the high-quality correct image 13B is generated. Compared with the case where a learning model that has not been learned is used as the initial state for learning, fine-tuning can generate a model at a very high speed, which is effective in improving the processing throughput.

[0158] In addition, fine-tuning is also suitable for creating a learning model that combines multiple purposes, such as using a learning model that has been learned for the purpose of noise removal as the initial state and using a learning paired image for the purpose of improving aberration. In other words, the relearned learning model can perform image transformation processing for both noise removal and aberration improvement, for example.

[0159] Fig.19 It means that for Fig.11 A table of model fit areas using the same expression method, illustrating how the values ​​in the table change due to relearning.

[0160] Fig.19 The table combines the initial learning model (first to third columns) as the initial state, the model representing the result of the first relearning (fourth and fifth columns), and the model representing the result of the second relearning (sixth and seventh columns) into one table. Since the identifiers of the model suitable areas of the three models are the same, they are only recorded in the first column.

[0161] The second, fourth, and sixth columns of "Judgment" are flags indicating whether the area is a model-appropriate region ((=1) or (=0)).

[0162] The third, fifth, and seventh columns of "Storage Destination" indicate the names of directories storing learning paired images used when fine-tuning and adjusting the learning model. The learning paired images stored in the directories are used as learning data for re-learning.

[0163] The initial learning model indicates that the model fits the area [-5, 5). The result of the first learning is expanded to the model fit area [-15, 5). The result of the second learning is expanded to the model fit area [-40, 5). In addition, the paired learning images used in the first learning are stored in the directories "A003, A004". The paired learning images used in the second learning are stored in the directories "A005 to A009".

[0164] For example, when the offset size of the low-quality image 11 that is newly processed is in the area [-15, -5), a part of the interval [-15, -5) is not suitable outside the interval in the initial learning model, but if the result of the first learning is used, all of them become suitable within the interval.

[0165] Fig. 20 This is a graph showing the state of occurrence of errors in the low-quality image 11 before processing by the image conversion unit 12 (before image quality improvement).

[0166] Fig.21 This is a graph showing the state of occurrence of errors in the high-quality equivalent image 13 after processing (after image quality improvement) by the image conversion unit 12 .

[0167] The horizontal axis of the graph represents the offset obtained using the high-quality equivalent image 13. The vertical axis of the graph represents the error between the offset obtained using the low-quality image 11 or the high-quality equivalent image 13 and the offset obtained using the high-quality correct image 13B. In addition, the learning model 14 is created using paired images for learning with an offset of -40 to +5 in the upper and lower layers.

[0168] To generate Fig. 20 and Fig.21 The number of images measured is 200 for each chart. Fig.21 and Fig. 20 By comparison, it can be seen that the deviation of the error is smaller in the total offset. A smaller deviation of the error means improved accuracy, and it can be seen that the measurement accuracy is improved by the image conversion unit 12. In addition, the horizontal axis is set to the deviation amount obtained using the high-quality equivalent image 13 instead of the deviation amount obtained from the high-quality image because, in the judgment process of the learning model being unsuitable, the high-quality image cannot be used for the reason described in the problem of obtaining the high-quality image.

[0169] exist Fig.21 In the example, the learning model was generated using paired learning images with an offset of -40 to +5 between the upper and lower layers. The offset between the upper and lower layers is caused by a problem in some manufacturing process, and in actual use, it is difficult to prepare images with such a wide range of offsets as paired learning images.

[0170] Fig. 22 This is a diagram showing the state of measurement error when image transformation is performed using a learning model in which the offset amount of the learning paired image is set to the model fit region [-5, 5).

[0171] exist Fig. 22 In the interval of the model suitable area, the deviation of the error becomes Fig.21 The measurement accuracy is considered to be improved when the values ​​are approximately the same. However, in the case of an image that is offset to the negative side from the model fit region, the error becomes larger as it moves away from the model fit region. In this way, if the offset of the low-quality image 11 is included in the model fit region, the deviation of the error is small and the inappropriate movement of the circuit pattern is also small. If the offset of the low-quality image 11 is not included in the model fit region, the deviation of the error is large and the inappropriate movement of the circuit pattern is also large.

[0172] Fig.23 This is a graph showing the error occurrence state based on the result of the first learning.

[0173] Through fine-tuning, the model fit region is expanded to the interval [-15, 5), and it can be seen that the error (vertical axis) in this interval becomes smaller. Based on the model fit region [-15, 5) of the learning model 14, the unfit detection unit 15 determines that the subsequent model is fit.

[0174] Fig.24 It is a graph showing the error occurrence state based on the result of the second learning.

[0175] Through fine-tuning, the model fit region is expanded to the interval of [-40, 5), and it can be seen that the error (vertical axis) in almost the entire interval is reduced. Based on the model fit region [-40, 5) of the learning model 14, the unfit detection unit 15 determines that the subsequent model is fit.

[0176] In addition, if Fig. 9 As shown, the offset of the obtained paired images for learning is concentrated in one place or becomes rough. If many images are concentrated in the area of ​​the identifier of a certain model suitable area, the weight of the area becomes large, which may lead to unbalanced re-learning.

[0177] Therefore, it is preferred Fig.11 The quantized width of the model fit region in the table expressing the model fit region is constant. Thus, by fixing the number of paired images for learning in each model fit region, it is possible to prevent such imbalance from occurring.

[0178] In the unsuitable detection unit 10 of the present embodiment described above, the unsuitable detection unit 15 determines whether the learning model 14 used in the transformation from the low-quality image 11 to the high-quality equivalent image 13 is suitable for the low-quality image 11. Based on the determination of the unsuitable detection unit 15, the unsuitable reporting unit 16 notifies that the high-quality equivalent image 13 does not become the target high-quality image because the learning model 14 is not suitable for the low-quality image 11 of the input data.

[0179] Thus, even if the high-quality equivalent image 13 is image-improved, the user can understand that the model is not suitable, and thus it is possible to prevent erroneous judgment based on the image content of the high-quality equivalent image 13 .

[0180] The unsuitability countermeasure unit 20 also takes countermeasures such as adding a learning target image corresponding to the low-quality image 11 judged as unsuitable input data to the learning data and performing re-learning. In this way, a learning model 14 suitable for the low-quality image 11 is created.

[0181] In this way, by relearning the learning model 14, it is possible to sequentially expand the model applicable region corresponding to the learning model 14. For example, when the learning model is applicable to noise but not to aberration, the existing model relearning unit 26 can generate a learning model 14 applicable to both by relearning using both the noise image and the aberration image.

[0182] In addition, the present invention is not limited to the above-described embodiments, and includes various modified examples. For example, the above-described embodiments are examples described in detail to explain the present invention in an easy-to-understand manner, and are not limited to necessarily having all the described structures.

[0183] Furthermore, a part of the structure of a certain embodiment may be replaced with the structure of another embodiment, and the structure of another embodiment may be added to the structure of a certain embodiment.

[0184] In addition, other structures can be added, deleted, or replaced with a part of the structure of each embodiment. In addition, the above-mentioned structures, functions, processing parts, processing units, etc. can also be implemented in part or in whole by hardware, such as by designing with integrated circuits.

[0185] Furthermore, each of the above-described structures, functions, and the like may also be realized by software by a processor interpreting and executing a program that realizes each function.

[0186] Programs, tables, files and other information that implement various functions can be pre-stored in memory, hard disk, SSD (Solid State Drive) and other recording devices, or IC (Integrated Circuit) card, SD card, DVD (Digital Versatile Disc) and other recording media. In addition, the cloud can also be effectively used.

[0187] In addition, the control lines and information lines are those considered necessary for explanation, and may not necessarily represent all the control lines and information lines on the product. In fact, it can be considered that almost all the structures are connected to each other.

[0188] Furthermore, the communication means for connecting the devices is not limited to the wireless LAN, and may be changed to a wired LAN or other communication means.

[0189] Explanation of symbols

[0190] 10 Unsuitable detection unit (unsuitable detection device),

[0191] 10B Model Learning Department,

[0192] 11 low quality image (image before transformation),

[0193] 12Image conversion unit,

[0194] 12B weight correction unit,

[0195] 13 high quality equivalent images (transformed images),

[0196] 13B high quality correct answer images (learning images),

[0197] 14Learning Model,

[0198] 15 Not suitable for detection department,

[0199] 16 Not suitable for reporting department,

[0200] 20 Non-conformity countermeasures department (non-conformity detection device),

[0201] 21 Countermeasures DB,

[0202] 22Countermeasures Search Department,

[0203] 23 Countermeasures and Methods Prompt Department,

[0204] 24Re-learning data input unit,

[0205] 24b Relearning Data Collection Department,

[0206] 25Use model change department,

[0207] 26Existing model re-learning department,

[0208] 27 New Model Learning Department,

[0209] 30 camera device,

[0210] 31Electron microscope,

[0211] 32X-ray tomography device,

[0212] 33 captured image storage unit,

[0213] 40 Image Utilization Department,

[0214] 41 Image observation processing unit,

[0215] 42 Image measurement processing unit,

[0216] 43 Image classification processing unit,

[0217] 50 controls the display unit.

Claims

1. A non-suitability detection device, characterized in that: have: An image transformation unit that transforms an input pre-transformation image into a post-transformation image using a learning model; an incompatibility detection unit configured to detect whether the pre-transformation image is incompatible with the learning model; The Non-Fitness Reporting Unit, which reports detected non-fits; and a storage unit for storing, as a model fit region, a distribution of evaluation values ​​of the learning images used in a learning phase of the learning model in correspondence with the learning model; The unsuitability detecting unit determines that the learned model is unsuitable when the evaluation value of the pre-conversion image is not within the range of the model suitability region.

2. The unsuitability detection device according to claim 1, characterized in that: The unsuitability detection device further comprises a usage model changing unit, When the unsuitability detection unit detects unsuitability, the usage model change unit searches from the storage unit for other of the learning models corresponding to the model suitable area suitable for the evaluation value of the image before transformation, and controls the image transformation unit to use the other of the learning models for the transformation processing of the image before transformation.

3. The unsuitability detection device according to claim 1, characterized in that: The unsuitability detection device further comprises an existing model relearning unit, When the unsuitability detection unit detects unsuitability, the existing model relearning unit relearns the learning model that has become unsuitable based on the additional learning image, thereby expanding the model suitable area of ​​the learning model.

4. The unsuitability detection device according to claim 3, characterized in that: The unsuitability detection device further comprises a countermeasure method prompting unit and a relearning data input unit. The countermeasure presenting unit presents a countermeasure for inappropriateness including an additional method of photographing the learning image. The re-learning data input unit receives input of the additional learning image captured by the presented imaging method.

5. The unsuitability detection device according to claim 3, characterized in that: The unsuitability detection device further comprises a relearning data collection unit, The re-learning data collection unit operates the imaging device according to a preset operation procedure, thereby accepting input of the additional image for learning that has been captured.

6. A method for detecting unsuitability, characterized in that: Unsuitable detection devices have: An image transformation unit that transforms an input pre-transformation image into a post-transformation image using a learning model; an incompatibility detection unit configured to detect whether the pre-transformation image is incompatible with the learning model; The Non-Fitness Reporting Unit, which reports detected non-fits; and a storage unit for storing, as a model fit region, a distribution of evaluation values ​​of the learning images used in a learning phase of the learning model in correspondence with the learning model; The unsuitability detecting unit determines that the learned model is unsuitable when the evaluation value of the pre-conversion image is not within the range of the model suitability region.

Images