Image evaluation device and method

Abstract

The present invention provides an image evaluation device and method, which can detect unknown defects and prevent misidentification of machine learning models. The image evaluation device of the present invention uses a machine learning-based identifier to identify defect information in a defect image of an electronic device, and is characterized by comprising: an image storage unit that stores defect images of electronic devices; a defect area storage unit that stores defect area information in defect images; an identifier that identifies defect information through machine learning; an image extraction unit that extracts image information of interest that the identifier is concerned about during the process of defect image recognition processing; and an evaluation unit that compares the image information of interest with the defect area information to evaluate the recognition possibility of the defect image.

Description

Technical Field

[0001] The present invention relates to an image evaluation device and method thereof that is realized by comparing a focus area of ​​a machine learning model in a defect image with position information of a defect object. Background Art

[0002] In recent years, in the field of semiconductor inspection and the like, image analysis technology has been used to extract feature quantities from images and compare and collate them with information registered in advance in a database or the like to discriminate objects.

[0003] As an example of this image analysis technology, there is known a method described in Patent Document 1. According to Patent Document 1, it is possible to output an output image in which a specific pattern is emphasized from an input image by applying a machine learning method.

[0004] In addition, as specific machine learning algorithms for machine learning means for distinguishing objects, neural networks and support vector machines are known. However, the recognition accuracy of these methods varies greatly depending on what feature quantity is selected, so the method of selecting the feature quantity is important.

[0005] In this regard, in recent years, a deep learning machine called CNN (Convolutional Neural Network) has been developed and has attracted attention. CNN is a machine learning machine. The deep learning model automatically learns the features of an image and performs the extraction of object images included in the image, object identification, image classification, etc.

[0006] Using CNN, it is possible to automatically extract the feature quantity required in machine learning such as previous support vector machines from the learning data, and to demonstrate very high image analysis performance.

[0007] Prior art literature

[0008] Patent Literature

[0009] Patent Document 1: Japanese Patent Application Publication No. 2005-34211 Summary of the invention

[0010] Problems to be solved by the invention

[0011] Thus, CNN has high recognition performance, but on the other hand, it also reacts strongly to images of unknown types other than the learned types, and has a problem of often making false recognitions with high confidence.

[0012] Specifically, for example, when CNN is applied to a semiconductor production line, some defects found in the semiconductor production line have a low frequency of occurrence, so it is difficult to collect and learn a large amount of them.

[0013] Therefore, there is a problem that there is a risk of missing unknown defects that are not included in the teaching data.

[0014] In this way, when machine learning is applied to semiconductor image inspection, there is a risk that the learned model will react strongly to unknown defect images that are not included in the teaching data and misidentify with high confidence. Therefore, there is a possibility of missing fatal defects and causing a decrease in yield. Therefore, a means of detecting unknown defects is required.

[0015] According to the above, the present invention aims to provide an image evaluation device and method that can detect unknown defects and prevent misrecognition of machine learning models.

[0016] Technical solutions to solve problems

[0017] According to the above, the present invention adopts "an image evaluation device that uses a machine learning-based identifier to identify defect information in a defect image of an electronic device, characterized in that it includes: an image storage unit that stores defect images of electronic devices; a defect area storage unit that stores defect area information in defect images; an identifier that identifies defect information through machine learning; an image extraction unit that extracts image information of interest that the identifier is concerned about during the process of defect image recognition processing; and an evaluation unit that compares the image information of interest with the defect area information to evaluate the possibility of identifying the defect image."

[0018] In addition, the present invention adopts "an image evaluation device that uses a machine learning-based identifier to identify defect information in a defect image of an electronic device, characterized in that it includes: an image storage unit that stores defect images of electronic devices; multiple identifiers that identify defect information through machine learning; an image extraction unit that extracts image information of interest that multiple identifiers are concerned about during the defect image recognition process; and an evaluation unit that compares multiple image information of interest to evaluate multiple identifiers."

[0019] In addition, the present invention adopts "an image evaluation method, which uses a machine learning-based identifier to identify defect information in a defect image of an electronic device, characterized in that: the defect image of the electronic device is saved, the defect area information in the defect image is saved, and during the recognition process of the defect image, the focus image information that the identifier is concerned about is extracted, and the focus image information is compared with the defect area information to evaluate the recognition possibility of the defect image."

[0020] In addition, the present invention adopts "an image evaluation method, which uses multiple machine learning-based identifiers to identify defect area information in defect images of electronic devices, characterized in that: the defect image of the electronic device is saved, and during the recognition process of the defect image, multiple image information of interest to multiple identifiers is extracted, and multiple identifiers are evaluated by comparing multiple image information of interest."

[0021] Effects of the Invention

[0022] According to the present invention, unknown defects can be detected and misidentification of machine learning models can be prevented. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a diagram showing an embodiment of an image evaluating device including an image evaluating unit.

[0024] Figure 2 It is a diagram showing an embodiment of an image region comparison unit based on center point comparison.

[0025] Figure 3 It is a diagram showing an embodiment of an image region comparison unit based on rectangular region comparison.

[0026] Figure 4 It is a diagram showing an embodiment of an image evaluation device including a defect detection unit.

[0027] Figure 5 This is a diagram showing an embodiment of an image evaluation device that compares multiple machine learning models.

[0028] Figure 6 It is a diagram showing an embodiment of an image evaluation device including a model re-learning unit.

[0029] Figure 7 It is a diagram showing an example of a processing flow of an image evaluation method.

[0030] Figure 8 It is a diagram showing an embodiment of an image evaluating device having an evaluation result display unit.

[0031] Fig. 9 It is a figure which shows the display example in the evaluation result display part.

[0032] Fig.10 This is a diagram for explaining an example of a semiconductor inspection system.

[0033] Fig.11 This is a schematic diagram of a scanning electron microscope.

[0034] Fig.12 It is a diagram that represents the learning method of the machine learning model. DETAILED DESCRIPTION

[0035] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0036] The image evaluation device of the present invention can be applied to a wide range of technical fields for processing images, but in the embodiments of the present invention shown below, an image evaluation device and method for preventing misidentification of unknown defects in semiconductor inspection using machine learning are shown as an example. In addition, as a specific example, an example of detecting unknown defects using a learned machine learning model and regional information of defective objects is shown.

[0037] For this purpose, in this manual, first use Fig.10 , Fig.11 Explain the technology that is the prerequisite for semiconductor inspection and then use it Figures 1 to 9 The embodiments are described.

[0038] Below, use Fig.10 , Fig.11 The present invention describes an apparatus and a measurement inspection system having an unknown defect detection function in semiconductor inspection using machine learning. More specifically, the present invention describes an apparatus and a system including a critical dimension-scanning electron microscope (CD-SEM or SEM) as a measurement device.

[0039] In addition, in the following description, a charged particle beam device is illustrated as a device for forming an image, and an example of using a SEM is described as one of the methods, but it is not limited to this. For example, a focused ion beam (FIB) device that forms an image by scanning an ion beam on a sample may also be used as a charged particle beam device. However, in order to measure a gradually refined pattern with high precision, a very high magnification is required, so it is preferable to use an SEM that is generally superior to a FIB device in terms of resolution.

[0040] Fig.10 This is a schematic diagram of a semiconductor inspection system in which a plurality of measuring or inspection devices are connected to a network. In this system, a CD-SEM 2401 that measures the pattern dimensions of a semiconductor wafer or a photomask, etc., and a defect inspection device 2402 that obtains an image by irradiating an electron beam to a sample and extracts defects based on comparison with a pre-registered reference image are connected to the network NW.

[0041] In addition, the network NW is connected to a condition setting device 2403 for setting measurement positions and measurement conditions on the design data of the semiconductor device, a simulator 2404 for completing simulation results of a pattern based on the design data of the semiconductor device and the manufacturing conditions of the semiconductor manufacturing device, and a storage medium 2405 for storing design data that registers the layout data and manufacturing conditions of the semiconductor device.

[0042] The design data is expressed in, for example, a GDS format or an OASIS format and stored in a prescribed format. In addition, the design data may be of any type as long as the software that displays the design data can display the format and process it as graphic data. In addition, the storage medium 2405 may also be built into a control device of a measuring device or an inspection device, or a condition setting device 2403 or a simulator 2404.

[0043] In addition, the CD-SEM 2401 and the defect inspection device 2402 have respective control devices to perform the control required for each device, but the functions of the above-mentioned simulator and the setting functions of measurement conditions, etc. may also be installed in these control devices.

[0044] In the SEM, an electron beam emitted from an electron source is focused by a multi-stage lens, and the focused electron beam is scanned one-dimensionally or two-dimensionally on a sample by a scanning deflector.

[0045] The secondary electrons (SE) or backscattered electrons (BSE) emitted from the sample due to the scanning of the electron beam are detected by the detector and stored in a storage medium such as a frame memory in synchronization with the scanning of the above-mentioned scanning deflector. The image signals stored in the frame memory are accumulated by the computing device installed in the control device. In addition, the scanning performed by the scanning deflector can be performed in any size, position and direction.

[0046] The control and the like as described above are performed by the control device of each SEM, and the image and signal obtained as a result of the electron beam scanning are sent to the condition setting device 2403 via the network NW. In addition, in this example, the control device for controlling the SEM and the condition setting device 2403 are described as separate bodies, but the present invention is not limited thereto, and the control and measurement processing of the device may be performed together by the condition setting device 2403, or the control and measurement processing of the SEM may be performed together by each control device.

[0047] In addition, the condition setting device 2403 or the control device stores a program for executing measurement processing, and the measurement or calculation is performed according to the program.

[0048] In addition, the condition setting device 2403 has a function of generating a program (recipe) for controlling the operation of the SEM based on the design data of the semiconductor, and functions as a recipe setting unit. Specifically, the position for performing the necessary processing for the SEM, such as the required measurement point, the automatic focus, the automatic astigmatism correction, and the addressing point is set on the design data, the contour line data of the pattern, or the design data after the simulation, and based on the setting, a program for automatically controlling the sample stage and the deflector of the SEM is generated.

[0049] Fig.11 This is a schematic diagram of the structure of a scanning electron microscope. An electron beam 2503 extracted from an electron source 2501 by an extraction electrode 2502 and accelerated by an accelerating electrode (not shown) is focused by a focusing lens 2504, which is a form of a focusing lens, and then scanned one-dimensionally or two-dimensionally on a sample 2509 by a scanning deflector 2505. The electron beam 2503 is decelerated by a negative voltage applied to an electrode built into a sample stage 2508, and is focused by the lens action of an objective lens 2506 and irradiated onto a sample 2509.

[0050] When the electron beam 2503 irradiates the sample 2509, electrons 2510 such as secondary electrons and backscattered electrons are emitted from the irradiation site. The emitted electrons 2510 are accelerated toward the electron source due to the acceleration effect of the negative voltage applied to the sample, and collide with the conversion electrode 2512 to generate secondary electrons 2511.

[0051] The secondary electrons 2511 emitted from the conversion electrode 2512 are captured by the detector 2513, and the output I of the detector 2513 changes according to the amount of secondary electrons captured. The brightness of the display device (not shown) changes accordingly with the output I. For example, when forming a two-dimensional image, the image of the scan area is formed by synchronizing the deflection signal output to the scanning deflector 2505 with the output I of the detector 2513. In addition, Fig.11 The scanning electron microscope shown as an example in FIG. 1 includes a deflector (not shown) for moving the scanning area of ​​the electron beam.

[0052] in addition, Fig.11 The example of the example in which the electrons emitted from the sample are first transformed by the transformation electrode and then detected is described, but it is of course not limited to such a structure. For example, a structure in which the detection surface of the electron multiplier tube or the detector is arranged on the orbit of the accelerated electrons can also be adopted. The control device 2514 controls each structure of the scanning electron microscope and has a function of forming an image based on the detected electrons and a function of measuring the pattern width of the pattern formed on the sample based on the intensity distribution of the detected electrons called line profile.

[0053] Example 1

[0054] The image evaluation device and method of the first embodiment of the present invention are applied to the semiconductor inspection device described above. Figure 1 An embodiment of an image evaluation device having the function of detecting unknown defects is shown in FIG.

[0055] Figure 1 The diagram shows an embodiment of an image evaluation device 10 having an image evaluation unit 1 . The image evaluation unit 1 inputs a defect image 101 and defect area information 105 , and obtains an evaluation result 106 by referring to a machine learning model 102 .

[0056] The image evaluation device 10 is constructed using computer system software, but when schematically showing the internal processing functions of the image evaluation unit 1 or the intermediate products generated during the processing, it can be considered to include a focus area detection unit 103, a detected focus image 104, and an image area comparison unit 20.

[0057] Specifically, in the image evaluation unit 1, the defect image 101 and the machine learning model 102 are input to the image evaluation unit 1, and the attention area detection unit 103 generates the area information in the image that the machine learning model 102 pays attention to as the attention image 104. Then, the generated attention image 104 and the defect area information 105 are compared by the image area comparison unit 20, and the evaluation result 106 is output.

[0058] The image evaluation unit 1 can be built into the control device 2514, or the image processing can be performed by a built-in computing device, or an external computing device (such as a CPU) can be used via a network. Fig.10 The condition setting device 2403) shown as an example in the figure performs image evaluation.

[0059] in addition, Figure 1 In the figures, etc., data used as input, data used as intermediate products, or data used as final products are stored in some form of storage unit although not explicitly shown in the figures.

[0060] In the attention area detection unit 103, for the defect image 101, a feature map of the image is generated using the parameters of the machine learning model 102. When realizing the function of the attention area detection unit 103, the document "Zhou, Bolei, et al. "Learning deep features for discriminative localization." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2016." can be used.

[0061] The Zhou document relates to visualization of attention areas in an image by a CNN, where the CNN calculates a weighted sum of multiple feature maps extracted from the image and outputs it as an attention image for each pixel. Specifically, the weights of the neurons associated with these feature maps are used to calculate the weighted sum of the feature maps, and the result after resizing to the original image size is output as an attention image 104.

[0062] Thus, the attention image 104 includes information on the attention level of each pixel in the defect image 101. The area to which the machine learning model 102 reflects strongly has a high attention level, and the area to which the machine learning model 102 reflects weakly has a low attention level. The attention image 104 can be used to explain the attention area that is the basis for the recognition result generated by the machine learning model 102.

[0063] On the other hand, when considering the defect area information 105 as another input in the image area comparison unit 20, the position information of the defect object in the image is included. For example, it can be a rectangular area surrounding the defect object, or it can be a mask image along the edge of the defect object. The generation of the defect area information 105 can be done by an engineer visually checking the defect image and manually drawing it, or it can be generated by the defect detection unit 30 described later.

[0064] Figure 2 2 is a diagram showing an example of the image area comparison unit 20. Threshold processing 201 is performed on the attention image 104. For example, only pixels whose attention is greater than a certain threshold are extracted. Then, the center point of the extracted pixel is calculated by center calculation 202. Similarly, the center point is also calculated based on the defect area information 105. Using the two calculated center points, the distance between the two center points is calculated by center distance calculation 203, and the center distance 204 is output.

[0065] In the center calculation 202, the average value of each pixel to be calculated may be calculated, or a weighted sum with weights added may be calculated. Examples of weights include the degree of attention of the pixel or the degree of confidence that the pixel is a defect.

[0066] Regarding the obtained center distance, if the defect type of the defect image 101 is a type included in the teaching data, the machine learning model 102 recognizes the defect object of the same type and focuses on its area, so it can be considered that the value of the above-mentioned center distance 204 calculated by the image area comparison unit 20 is small. On the other hand, regarding the obtained center distance, in the case of an unknown defect not included in the teaching data, the deviation between the area focused on by the machine learning model 102 and the defect area is large, so it can be considered that the value of the center distance 204 is large. Because of this tendency regarding the center distance, the center distance 204 can be directly used as the evaluation result 106 of the image evaluation device 1. Alternatively, the comparison result of the center distance 204 with a certain threshold (for example, OK and NG) can also be output as the evaluation result 106.

[0067] Figure 2 In the above, as a determination method of the image area comparison unit 20, a phenomenon is utilized that a known defect of a type included in the teaching data and an unknown defect not included in the teaching data have a difference in center distance (center point) to identify them.

[0068] Relative to Figure 2 The image area comparison based on the center point comparison is as another embodiment 1 Figure 3 Perform image region comparison based on rectangular region comparison. Figure 3 Will Figure 2 The center calculation in the image area comparison unit 20 is replaced by rectangle calculation.

[0069] Figure 3 In the rectangle calculation 205, a rectangle including the target image area (interest area or defect area) is estimated. The rectangle repetition degree 207 is calculated by the rectangle repetition degree calculation 206 using the two estimated rectangles. As an example of the rectangle repetition degree 207, the ratio of the repeated area to the total area can be cited. Figure 2 The center distance 204 in can be used as an indicator for unknown defect judgment, so it can be directly output as the evaluation result 106, or the comparison result with the threshold value can be output as the evaluation result 106.

[0070] In the above description of Embodiment 1, an embodiment of the image region comparison unit 20 based on comparison of center points or comparison of rectangles is described. In addition, it is also possible to adopt methods such as comparison of irregular regions, comparison of region areas, comparison of region shapes, comparison of representative points (for example, using the maximum value of the attention degree as the representative point), etc. In addition, a method of combining multiple comparison methods can also be considered.

[0071] pass Figure 1 , 2, 3, the final evaluation result 106 includes one or more indicators output by the image area comparison unit 20, and the results calculated based on the indicators (for example, when the value of the indicator exceeds a certain threshold, it is judged as an unknown defect).

[0072] Figure 8 Yes means Figure 1 FIG. 1 is a diagram showing a configuration example in which the image evaluation device 10 further includes an evaluation result display unit 5. It is preferable to perform the following on the evaluation result display unit 5. Fig. 9 The image is displayed in the form shown as an example.

[0073] according to Fig. 9 In the evaluation result display example in FIG. 5 , a defect image 101 , defect region information 105 , a degree of attention image 104 , and an evaluation result 106 in which these images are superimposed are displayed on the display screen of the evaluation result display unit 5 .

[0074] In this case, in the evaluation result 106 , a region R1 indicated by diagonal lines is a defect region, a region R2 indicated by diagonal lines is a focus region detected from the focus image, and a region R3 is a region where both overlap.

[0075] If the area R3 is small, it means that the deviation between the focus area of ​​the machine learning model 102 and the defect area information 105 is large, and it can be considered that the defect image 101 is likely to be an unknown defect. Fig. 9 The evaluation result display unit 5 shown provides a deeper understanding of the evaluation result 106 .

[0076] In addition, through the above display, it is known in advance that the focus area of ​​the machine learning model 102 is correct, and when the defect area information 105 is visually produced by an engineer, the difference between the defect area marked by the engineer and the true value can be known based on the evaluation result 106, so it is believed that the image evaluation device can also be used for the education and ability judgment of engineers.

[0077] Through the above, it is possible to realize the image evaluating device including the image evaluating unit 1 .

[0078] The image evaluation device 10 of the present invention mainly relates to a device that uses a machine learning model and defect area information to evaluate an unknown defect image and determine whether it can be identified. It is constituted as "an image evaluation device that uses a machine learning-based identifier 102 to identify defect information (type, location, etc.) in a defect image of an electronic device, and is characterized in that it includes: an image storage unit that stores a defect image 101 of the electronic device, a defect area storage unit that stores defect area information 105 in the defect image, an identifier 102 that identifies defect information through machine learning, an image extraction unit 103 that extracts image information of interest 104 that the identifier pays attention to during the defect image recognition process, and an image evaluation unit 20 that compares the image information of interest with the defect area information to evaluate the possibility of identifying the defect image."

[0079] In addition, the image evaluation unit 20 (201, 202, 203) in the image evaluation device 10 of the present invention is constructed as "an image evaluation device that calculates one or more focus coordinates from the image area that the machine learning identifier 102 focuses on, calculates one or more defect coordinates based on the defect area information, and compares the distances between the coordinates to evaluate the possibility of recognizing the defective image" as an example of its processing.

[0080] Furthermore, the image evaluation unit 20 (201, 202, 203) is constructed as "an image evaluation device, the image evaluation unit (201, 202, 203) calculates a representative focus area from the image area that the machine learning identifier focuses on, calculates the defect area from the defect area information, calculates the repetition between the areas, and evaluates the possibility of recognizing the defect image" as an example of its processing.

[0081] In addition, the image evaluation device 10 of the present invention has an image display unit (evaluation result display unit 5) for displaying the evaluation result 106 of the image evaluation unit (image area comparison unit 20), and displays defect area information in the defect image, or at least one of a normal image corresponding to the defect image or design data corresponding to the defect image on the image display unit.

[0082] Example 2

[0083] In Figure 1 In the representative embodiment 1, the premise is that the defect area information 105 is obtained by some method.

[0084] In contrast, in Example 2, the Figure 4 A specific method for obtaining the defect area information 105 is described below. Figure 4 , having a defect detection unit 30, which generates defect area information 105 in the image from a defective image 101 and a normal image 301 or design data 302.

[0085] Here, an embodiment of the defect detection unit 30 using the normal image 301 is described. The normal image 301 is a reference image that has the same photographed pattern as the defect image 101 but does not include a defective object. In addition, it is also possible to synthesize photographed images of multiple normal patterns and use the generated image as the normal image 301. In the defect detection unit 30, the difference between the defect image 101 and the normal image 301 is calculated, and the area with a larger difference is determined as a defect area, and the defect area information 105 is generated.

[0086] In addition, it is also possible to consider that the defect detection unit 30 uses the contour information such as the edge extracted from the defect image 101, compares it with the design data 302, detects the defect area based on the deviation between the two, and generates the defect area information 105. In addition, the normal image 301 is obtained when the image is correctly formed according to the design data 302, so it can be considered to be equivalent to the design data 302.

[0087] Example 2 further involves the generation of defect area information 105 in the structure of Example 1, and is constituted as "an image evaluation device, characterized in that it has a defect detection unit 30 that uses a defect image 101 and uses either or both of a normal image 301 corresponding to the defect image and design data 302 corresponding to the defect image to detect the defect area information 105, and a defect area storage unit stores the defect area information 105 from the defect detection unit 30."

[0088] Example 3

[0089] In the third embodiment, a method of generating a plurality of attention images from different viewpoints and comparing them for evaluation is described. The defect region information 105 of the first embodiment can be replaced with a second attention image generated from a different viewpoint.

[0090] about Figure 5 The image evaluation unit 1 of Example 3 shown as an example can be regarded as including a focus area detection unit 103 that refers to two machine learning models 102 and 107; two detected focus images 104 and 108; and an image area comparison unit 20 when schematically showing its internal processing functions or intermediate products generated during the processing.

[0091] Figure 5 In the image evaluation unit 1, a defect image 101 and two machine learning models 102 and 107 are input into the image evaluation unit 1, the attention image 104 and 108 of each model are detected by the attention area detection unit 103, the two attention images are compared by the image area comparison unit 20, and the comparison result is output as the evaluation result 106.

[0092] For example, one machine learning model 102 is used as a reference model, and another machine learning model 107 is used as an evaluation target model. In this case, the evaluation result 106 can evaluate the performance of the evaluation target model relative to the performance of the reference model.

[0093] in addition, Figure 5 Although two machine learning models are input to the image evaluation unit 1, it is also possible to consider using an image evaluation device 10 that inputs three or more machine learning models, compares their respective attention images, and outputs an evaluation result 106.

[0094] Example 3 adopts "an image evaluation device, which uses multiple machine learning-based identifiers (102, 107) to identify defect information (type, location, etc.) in a defect image of an electronic device, characterized by: an image storage unit for storing the defect image 101 of the electronic device, an image extraction unit (103) for extracting multiple focus image information that multiple identifiers focus on during the recognition process of the defect image, and an identifier evaluation unit (20) for comparing the multiple focus image information (104, 108) to evaluate the multiple identifiers."

[0095] Example 4

[0096] In Example 1, the evaluation result is finally obtained, and the user is prompted with information, such as by display, etc., but in Example 4, the evaluation result is finally obtained, and the user is prompted with information, such as by display, etc. Figure 6 This describes how to gradually build a more accurate machine learning model by reflecting the evaluation results in the machine learning model.

[0097] exist Figure 6 An embodiment of an image evaluation device 10 having a function of relearning a machine learning model is shown in FIG. Figure 6 In the image evaluation device 10 , a model re-learning unit 4 is additionally provided for obtaining learning information of the model using the evaluation result 106 outputted by the image evaluation unit 1 , and adjusting the machine learning model 102 .

[0098] The model relearning unit 4 is composed of the functions of teaching data adjustment 401, learning parameter adjustment 402, and model learning 403. In the model relearning unit 4, the teaching data adjustment 401 is first performed. For example, when the evaluation result 106 exceeds the threshold and an unknown defect image is detected, the unknown defect image is defined as a new defect type. Alternatively, for the unknown defect image, clustering processing is performed, and each generated cluster is defined as a new defect type and appended to the teaching data. In addition, when the recognition result of the machine learning model 102 is different from the label value of the image, and the evaluation result 106 indicates that it is not an unknown defect, there is a possibility of modifying the label value of the image. For example, label B is attached to an image whose true value is A. When the recognition result is A, and according to the evaluation result 106, the focus area of ​​the machine learning model 102 coincides with the defect area, there is a possibility that the label value of the image is wrong.

[0099] It is also possible to consider using the evaluation result 106 as a reference to perform learning parameter adjustment 402 .

[0100] For example, if it is known in advance that the defect image 101 is not an unknown defect, but the suspicion of an unknown defect is high according to the evaluation result 106, in order to improve the recognition performance of the machine learning model 102, the penalty for misrecognition of the defect image of this type is increased. Specifically, the weight of each type used in learning is adjusted. In addition, it is also possible to consider that if the deviation between the focus area of ​​the machine learning model 102 and the defect area is large, it is judged that the learning is insufficient and the number of learning is increased.

[0101] Then, the machine learning model 102 is relearned using the model learning 403 to generate the machine learning model 102 as an input to the image evaluation unit 1 .

[0102] in addition, Figure 6 In the model re-learning unit 4, it is depicted as including the teaching data adjustment 401 and the learning parameter adjustment 402, but these two steps are not necessarily required.

[0103] Embodiment 4 relearns the machine learning model 102, one example of which is to adjust the true value of the teaching data, using "an image evaluation device, characterized in that it has a teaching data adjustment unit (401) that adjusts the teaching data based on the comparison result of the image information of interest and the defect area information, and an identifier (102) that relearns based on the information from the teaching data adjustment unit."

[0104] Alternatively, another example of Example 4 adjusts the learning parameters by using "an image evaluation device, characterized in that it has a machine learning parameter adjustment unit that adjusts machine learning parameters based on the comparison results of the image information of interest and the defect area, and an identifier (102) that relearns based on the information from the machine learning parameter adjustment unit."

[0105] Example 5

[0106] Figure 7 The illustrated embodiment 5 is a flowchart describing a series of processing contents in the image evaluation method of the present invention.

[0107] According to this processing flow, the defect area detection process shown in the processing step S10 and the attention area detection process shown in the processing step S20 are performed in parallel, and when the processing results of both are complete, the next processing step S30 is executed to perform image area comparison processing. Furthermore, the model relearning process is then performed in the processing step S40 (the case of Example 4).

[0108] In addition, the processing other than the attention area detection processing S20 and the image area comparison processing S30 is an optional step. Regarding the above image evaluation method, part or all of the processing can be generated by a processing program running in a general-purpose CPU. In addition, it is also possible to consider making part or all of the processing a dedicated LSI or FPGA.

[0109] As Example 5 of the image evaluation method, "an image evaluation method (including the entire system of 1) is adopted, which uses a machine learning-based identifier (102) to identify defect information (type, location, etc.) in a defect image of an electronic device, characterized in that: the defect image of the electronic device (101) is saved, the defect area information in the defect image is saved (105), and during the recognition process of the defect image, the image information (103, 104) that the identifier pays attention to is extracted, and the image information of interest is compared with the defect area information to evaluate the recognition possibility of the defect image (20)".

[0110] It is possible to consider using the output value calculated by model 102 to determine whether defect image 101 is an unknown defect. For example, when model 102 is an image classification model, the output probability values ​​belonging to each category are determined to be unknown defects when the highest probability among them is still less than the threshold. However, according to the document "Nguyen, Anh, Jason Yosinski, and Jeff Clune. "Deep neural networks are easily fooled: High confidence predictions for unrecognizable images." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2015.", in most cases, there is a tendency for the probability value of the model output to be too high. Therefore, the performance of unknown defect discrimination based on thresholds is not high.

[0111] In addition, most existing methods use a constraint condition that the sum of the probability values ​​belonging to each class becomes 1.0 through the Softmax function. In this way, the probability value belonging to a certain class will affect the probability values ​​belonging to other classes. Therefore, the probability value is not the absolute probability value belonging to a certain class, but the relative probability value of all learned classes. It is difficult to distinguish unknown defects using threshold processing of relative probability. For example, using the three-class classification task of "bird, tiger, and fish" to identify images of cats. Cats are different from any of the three classes, so in the case of absolute probability, all three probability values ​​are low. However, in the case of relative probability, among the three classes, cats are relatively similar to tigers, so the probability value belonging to tigers is higher. In this case, it is difficult to identify cats as unknown classes through threshold processing.

[0112] In order to solve the above problems, a Fig.12 The model learning method shown. First, extract the feature value from the image in step S601. Using the extracted feature value, calculate the probability value belonging to each class in step S602. The probability value is an absolute probability value, which indicates the possibility of belonging to the class. Then, update the feature center point of the true value class of the image in step S603. After that, calculate the distance from the feature value of the image to the feature center point of each class in step S604. Using this distance, calculate the target probability value of each class in step S605.

[0113] Finally, in step S606, the parameters of the model are updated so that the probability value calculated in step S602 approaches the target probability value. In this way, steps S601 to S606 are repeated until the convergence condition of learning is satisfied, and the model is learned.

[0114] In the above description, step S603 of updating the feature center point is called before step S604 of calculating the distance between the feature value of the image and the feature center point is executed, but step S603 can also be placed after step S604. Alternatively, it is also possible to consider executing step S603 after performing steps S604 to S606 multiple times.

[0115] As an example of the image feature extraction step S601, a multi-layer neural network can be cited. Alternatively, a feature designed by a human can also be considered. Basically, a multi-dimensional image feature is extracted.

[0116] As an embodiment of step S602 of calculating the absolute probability value, a multi-dimensional image feature quantity is transformed into one dimension using a regression method such as multi-factor regression analysis or support vector regression, and then transformed into a range of 0.0 to 1.0 using a logical function such as Sigmoid. It is also possible to consider using a neural network to transform the multi-dimensional image feature quantity into a one-dimensional value.

[0117] As an embodiment of step S603, the difference between the image feature quantity and the feature center point of the true value class is calculated. After the difference is multiplied by a coefficient, it is added to the current feature center point. The coefficient by which the difference is multiplied can be set as a parameter set by the user, or it can be a value automatically determined according to the number of samples of the true value class.

[0118] Using the distance to the feature center point of each class calculated in step S604, the target probability value of each class is calculated in step S605. The longer the distance, the smaller the target probability value. For example, an exponential function of the distance can be considered. In most existing methods, a one hot vector is used in which the target probability value is "1.0 for the true value class and 0.0 for other classes".

[0119] In the present invention, instead of One Hot, a value that takes into account the distance from the feature center point may be used as the target probability value. In addition, in order to reduce the amount of calculation, it is also possible to consider setting the target value other than the true value class to zero or any determined value.

[0120] It is explained that when evaluating a defect image, whether it is an unknown defect is determined based on the probability value and threshold calculated by the model, but it is also possible to consider determining based on the distance between the image feature quantity extracted by the model and the feature center point of each type. Alternatively, the probability value output by the model can be used as a coefficient, multiplied by the attention image 104, and then the image area comparison unit 20 can be used to determine whether it is an unknown defect.

[0121] Description of Reference Numerals

[0122] 1: Image evaluation unit, 20: Image area comparison unit, 30: Defect detection unit, 40: Model relearning unit, 5: Evaluation result display unit, 101: Defect image, 102, 107: Machine learning model, 103: Interest area detection unit, 104, 108: Interest image, 105: Defect area information, 106: Evaluation result, 201: Threshold processing, 202: Center calculation, 203: Center distance calculation, 204: Center distance, 205: Rectangle calculation, 206: Rectangle repeatability calculation, 207: Rectangle repeatability , 301: normal image, 302: design data, 401: teaching data adjustment, 402: learning parameter adjustment, 403: model learning, S10: defect area detection processing, S20: focus area detection processing, S30: image area comparison processing, S40: model re-learning processing, S601: image feature extraction processing, S602: absolute probability value calculation processing, S603: feature center point update processing, S604: feature distance calculation processing, S605: target probability value calculation processing, S606: parameter update processing.

Claims

1. An image evaluation device, using a machine learning-based identifier to identify defect information in a defect image of an electronic device, characterized in that: include: An image storage unit for storing a defect image of the electronic device; A defect area storage unit for storing defect area information in the defect image; An identifier that identifies defect information through machine learning; An image extraction unit for extracting the image information of interest that the identifier is interested in during the process of the defect image recognition process; and an evaluation unit that compares the attention image information with the defect region information to evaluate the recognition possibility of the defect image, The evaluation unit calculates a representative focus area from the image area focused on by the machine learning-based identifier, calculates a defect area from the defect area information, and calculates the repetition between the areas to evaluate the recognition possibility of the defect image. The degree of overlap between the regions is a ratio of an overlapped area between the rectangular region of interest and the rectangular defect region to a total area of ​​the region of interest and the defect region.

2. The image evaluation device according to claim 1, wherein: A defect detection unit is provided for detecting defect region information using the defect image and using either or both of a normal image corresponding to the defect image and design data corresponding to the defect image. The defect area storage unit stores the defect area information from the defect detection unit.

3. The image evaluation device according to claim 1 or 2, characterized in that: The method includes a teaching data adjustment unit for adjusting teaching data based on a comparison result between the attention image information and the defect region information; and the identifier for performing re-learning based on information from the teaching data adjustment unit.

4. The image evaluation device according to any one of claims 1 to 3, characterized in that: It includes a machine learning parameter adjustment unit that adjusts the parameters of machine learning based on the comparison result of the focus image information and the defect area information; and the identifier that performs re-learning based on the information from the machine learning parameter adjustment unit.

5. The image evaluation device according to any one of claims 1 to 4, characterized in that: An image display unit is provided to display the evaluation result of the evaluation unit.

6. The image evaluation device according to claim 5, characterized in that: The image display unit displays defect region information in the defect image.

7. The image evaluation device according to claim 6, characterized in that: The image display unit displays at least one of a normal image corresponding to the defective image and design data corresponding to the defective image.

8. An image evaluation method, using a machine learning-based identifier to identify defect information in a defect image of an electronic device, characterized in that: saving a defect image of the electronic device, saving defect area information in the defect image, In the process of the defect image recognition processing, the image information of interest to the recognizer is extracted. Calculating a representative focus area from the image area focused on by the machine learning-based recognizer, calculating a defect area from the defect area information, and calculating the repetition between areas to evaluate the recognition possibility of the defect image, The degree of overlap between the regions is a ratio of an overlapped area between the rectangular region of interest and the rectangular defect region to a total area of ​​the region of interest and the defect region.

9. The image evaluation method according to claim 8, characterized in that: Using the defective image, and using either or both of a normal image corresponding to the defective image and design data corresponding to the defective image, defective region information is detected and stored.

10. The image evaluation method according to claim 8 or 9, characterized in that: The teaching data is adjusted based on a comparison result between the attention image information and the defect region information, and the identifier performs re-learning based on information from the adjusted teaching data.

11. The image evaluation method according to any one of claims 8 to 10, characterized in that: The parameters of the machine learning are adjusted based on the comparison result between the attention image information and the defect area information, and the identifier performs re-learning based on information from the adjusted machine learning parameters.

12. The image evaluation method according to any one of claims 8 to 10, characterized in that: The results of the evaluation are displayed.

Images