Image recognition system

Abstract

This invention provides an image recognition system. In an image recognition system that identifies shapes contained in an image, even when multiple shapes are reflected within the image, the success or failure of the prediction result for each shape can be determined individually. The image recognition system of this invention calculates the importance of each feature quantity for each object shape identified in the image, and compares the importance and statistical values ​​of each feature quantity for each object shape to determine whether the recognition result is correct.

Description

Technical Field

[0001] This invention relates to an image recognition system for identifying shapes contained in an image. Background Technology

[0002] In semiconductor and LCD panel production lines, defects occurring early in the process lead to wasted work in subsequent steps. Therefore, inspection processes are implemented at critical points in each process to ensure / maintain a given yield rate during manufacturing. These inspection processes include, for example, the use of scanning electron microscopy (SEM) for Critical Dimension SEM (CD-SEM) and Defect Review SEM.

[0003] In the inspection process, images captured by the inspection equipment are used to confirm the presence of defects and anomalies. In recent years, high-precision automated inspection has become possible using image processing models built from machine learning. However, the characteristics of the samples being inspected vary depending on the manufacturing process. Therefore, to maintain high inspection accuracy, the image processing model needs to be relearned on-site. In this case, a performance comparison evaluation between the relearned model and the existing model is required, as well as confirmation of whether the relearned model is functioning correctly. That is, a performance evaluation of the image recognition model is needed. Typically, performance evaluation is implemented using labeled data or third-party verification, but these are difficult to implement on-site from the perspectives of cost and time. Therefore, it is necessary to automatically evaluate the model's performance using unlabeled data.

[0004] As background technology in this field, there is, for example, technology like Patent Document 1. Patent Document 1, with the subject "providing a technology in an image classification apparatus and image classification method that can easily display information related to the reliability of the classification result in an image classification based on image feature quantities," describes the following technology: "Taking a defective image X classified into category A by any classification algorithm as the calculation object, the accuracy of its classification result is calculated. For each of the various feature quantities V1 to V8, the range of values ​​possessed by a typical image belonging to that classification category is pre-determined as a typical range. Among the feature quantities representing the calculation object image X, the feature quantities whose values ​​are within the typical range are voted on, and the ratio of the number of votes to the number of feature quantities is output as the accuracy." (See abstract).

[0005] Prior art literature

[0006] Patent documents

[0007] Patent Document 1: Japanese Patent Application Publication No. 2013-077127

[0008] The technical output of the aforementioned patent document 1 represents the accuracy and reliability of the classification results based on the classification algorithm (image recognition model). Therefore, since it is possible to determine whether the classification results of the image recognition model are correct, it can be considered that the model's performance can be automatically evaluated using unlabeled data.

[0009] However, the conventional image classification apparatus described in Patent Document 1 calculates feature values ​​for an entire input image. In other words, even if multiple defects exist in the image, it calculates the overall feature value for identifying all of those defects. Therefore, it identifies a collection of defects based on the feature value, rather than identifying individual defects. Consequently, it is difficult to determine the success or failure of the prediction results for each individual defect. Summary of the Invention

[0010] The present invention was made in view of the aforementioned problems, and its object is to provide a technique in an image recognition system for recognizing shapes contained in an image, which can determine the success or failure of individual prediction results for each shape even when multiple shapes are reflected in the image.

[0011] The image recognition system of the present invention calculates the importance of each feature quantity for each object shape identified in the image and for each type of feature quantity. It compares the importance and the statistics of each type of feature quantity for each object shape to determine whether the recognition result is correct.

[0012] -Invention Effects-

[0013] According to the image recognition system of the present invention, even when multiple shapes are reflected in an image, the success or failure of the prediction result for each shape can be determined individually. Other issues, structures, and effects beyond the above will become clear through the following description of embodiments. Attached Figure Description

[0014] Figure 1 This is a diagram that conceptually represents the structure of the image recognition system 100 according to Embodiment 1.

[0015] Figure 2 This is a block diagram representing the structure of the image recognition system 100.

[0016] Figure 3 This is a flowchart illustrating the operation of the image recognition system 100.

[0017] Figure 4 This is a flowchart showing the processing of the feature importance calculation unit 14.

[0018] Figure 5 This is an example of the result obtained by the feature importance calculation unit 14.

[0019] Figure 6 This is a block diagram representing the structure of the image recognition system 100.

[0020] Figure 7 This is a flowchart illustrating the process by which the statistics calculation unit 21 calculates the statistics.

[0021] Figure 8A This is an example of data stored in the feature importance database (DB) 20 in S114.

[0022] Figure 8B This is an example of data stored in the statistical information database (DB) 15 in S116.

[0023] Figure 9 It is a diagram that conceptually represents the determination of the correctness of prediction results based on distribution comparisons.

[0024] Figure 10 This is a block diagram illustrating the structure of the image recognition system 100 according to Embodiment 2.

[0025] Figure 11 This is a flowchart illustrating the operation of the image recognition system 100 in Embodiment 2.

[0026] Figure 12 This is an example of a GUI that displays the misidentification judgment results stored in the judgment result database (DB) 26 and the model evaluation results stored in the evaluation result database (DB) 28.

[0027] Figure 13 This is an example of a GUI that displays the comparison results stored in the comparison results database (DB) 30.

[0028] Figure 14 This is a flowchart illustrating the operation of the image recognition system 100 according to Embodiment 3.

[0029] Figure 15 This is an example of a GUI output by the display unit 31.

[0030] -Symbol Explanation-

[0031] 100: Image Recognition System

[0032] 2: Image Recognition Department

[0033] 6: Misidentification Decision Unit

[0034] 9: Sample

[0035] 10: Inspection device

[0036] 14: Feature Importance Calculation Section

[0037] 15: Statistical Information Database (DB)

[0038] 16: Comparison Section

[0039] 20: Feature Importance Database (DB)

[0040] 21: Statistics Calculation Department

[0041] 22: Model Database (DB)

[0042] 23: Model Input Section

[0043] 26: Judgment Result Database (DB)

[0044] 27: Model Evaluation Department

[0045] 28: Evaluation Results Database (DB)

[0046] 29: Model Comparison Section

[0047] 30: Compare the results database (DB)

[0048] 31: Display Section

[0049] 32: Model Monitoring Department

[0050] 33: Monitor the database (DB). Detailed Implementation

[0051] <Implementation Method 1>

[0052] Figure 1 This diagram conceptually illustrates the structure of the image recognition system 100 according to Embodiment 1 of the present invention. The image recognition system 100 includes an image recognition unit 2 and a misidentification determination unit 6, etc. The image recognition unit 2 obtains a prediction result 3 by performing prediction on the captured image 1. The misidentification determination unit 6 determines whether the prediction result 3 is correct.

[0053] As shown in prediction result 3, the image recognition unit 2 predicts the type and location of defects reflected in the camera image 1 separately, as in prediction results 4 and 5. The misidentification determination unit 6 takes the results predicted by the image recognition unit 2 as input and determines whether these results are correct. In other words, the misidentification determination unit 6 determines whether the predicted type of defect in each region of the prediction results is correct, outputting determination result 7 for prediction result 4 and determination result 8 for prediction result 5.

[0054] Figure 2This is a block diagram representing the structure of the image recognition system 100. (Using...) Figure 2 For use in Figure 1 The specific structure of the functions described in the text will be explained.

[0055] The inspection device 10 captures a photographic image 1 of the sample 9. The sample 9 is, for example, a semiconductor wafer. The inspection device 10 is equivalent to, for example, a defect inspection device based on a mirror electron microscope that images electrons from a mirror, an optical defect inspection device, etc.

[0056] Image recognition unit 2 performs defect inspection on the acquired camera image 1. Image recognition unit 2 extracts feature values ​​from camera image 1 and detects defects reflected in camera image 1 based on the extracted feature values. When multiple defects are reflected in camera image 1, image recognition unit 2 predicts these defects individually. Therefore, image recognition unit 2 possesses an image recognition model capable of predicting the type and location of defects. For example, an image recognition model composed of a CNN (Convolutional Neural Network) such as SSD (Single Shot Multibox Detector) or RetinaNet can be used.

[0057] The misidentification determination unit 6 includes a feature importance calculation unit 14, a statistical information database (DB) 15, and a comparison unit 16. Details of the processing content of each component unit will be described later.

[0058] The feature importance calculation unit 14 takes the prediction result 13 as input and calculates the feature importance based on the prediction result 13. The feature importance represents the importance of the feature extracted by the image recognition unit 2 relative to the prediction result. A specific example of the importance will be described later.

[0059] The Statistical Information Database (DB) 15 stores statistics related to the importance of features calculated by the Feature Importance Calculation Unit 14.

[0060] The comparison unit 16 compares the feature importance calculated by the feature importance calculation unit 14 with the statistics stored in the statistical information database (DB) 15. If the feature importance calculated by the feature importance calculation unit 14 is different from the statistics stored in the statistical information database (DB) 15, the comparison unit 16 determines the prediction result 13 to be incorrect and outputs it as the determination result 17.

[0061] Figure 3 This is a flowchart illustrating the operation of the image recognition system 100. The following is a description of... Figure 3 The steps are explained below.

[0062] In step S101, the inspection device 10 captures a photographic image 1 of the sample 9.

[0063] In step S102, the image recognition unit 2 performs image processing on the camera image 1 to predict the type and location of defects reflected in the camera image 1, and outputs the prediction result 13.

[0064] In step S103, the feature importance calculation unit 14 calculates the feature importance for each prediction result.

[0065] In step S104, the comparison unit 16 compares the feature importance calculated by the feature importance calculation unit 14 with the statistics stored in the statistical information database (DB) 15 to determine whether the prediction result is correct for each prediction result.

[0066] Reference Figure 4 Figure 8 provides a detailed explanation of the processing details of each component of the misidentification determination unit 6: the feature importance calculation unit 14, the statistical information database (DB) 15, and the comparison unit 16.

[0067] Figure 4 This is a flowchart illustrating the processing of the feature importance calculation unit 14. The following is a description of... Figure 4 The steps are explained below.

[0068] In step S105, the feature importance calculation unit 14 uses error backpropagation to calculate the differential of the feature map of the image recognition unit 2 with respect to the prediction result. Using this differential value, the importance of each feature map for the prediction result is calculated. The feature map retains the features extracted from the camera image 1. The processing of this step is shown in Equation 1.

[0069] [Mathematical Expression 1]

[0070]

[0071] In mathematical formula 1, y c，box_pre This is the score for level c (defect type) predicted by the image recognition unit 2, where box_pre represents the predicted location. A i，j，k Let i and j represent the pixel numbers in the vertical and horizontal directions of the feature map, respectively, and k represent the channel number. u and v represent the number of pixels in the vertical and horizontal directions of the feature map, respectively. z is u × v. In other words, through step S105, the derivative of the feature map with respect to the score value at level c and position box_pre is calculated, and the average value is calculated for each channel. The resulting α... k，c，box_preThis represents the importance of the feature quantity held by the feature quantity map for channel number k relative to the prediction result (level c, position box_pre). Since the importance is calculated for each channel, for example, in the case of 100 channels, 100 importance values ​​are calculated. In the case where multiple defects are reflected in image 1 and there are multiple prediction results, the importance is calculated for each prediction result. This importance indicates the degree of influence of the feature quantity type k on the recognition result.

[0072] In step S106, the feature importance calculation unit 14 weights the features held by the feature map using the importance calculated in step S105 and the region information of the prediction result. This process is shown in mathematical formula 2. k，c，box_pre The predicted regional information is obtained through mathematical formula 3. S k，c，box_pre This represents the importance of the region in each pixel of the feature map relative to the prediction result (level c, position box_pre). As region information for the prediction result, masks that set regions within the prediction result to 1 and other regions to 0, or template regions with pre-defined important regions, can also be used. If multiple defects are reflected in the image 1 and there are multiple prediction results, the processing of mathematical formula 2 is performed for each prediction result. In step S106, the feature importance calculation unit 14 can weight the features held in the feature map solely based on the importance calculated in step S105. G k，c，box_pre Or S k，c，box_pre This indicates the degree to which the image region affects the recognition result.

[0073] [Mathematical Expression 2]

[0074] G k，c，box_pre =α k，c，box_pre *A k *S k，c，box_pre

[0075] [Mathematical Expression 3]

[0076]

[0077] In step S107, the feature importance calculation unit 14 calculates the G value obtained in step S106. k，c，box_pre The importance of the feature quantity is calculated according to each channel number. The importance of the feature quantity can be calculated, for example, by using G... k，c，box_preThe method calculates the average value only for values ​​in the vicinity of the predicted result, and only for values ​​above a pre-set threshold. Through this processing, the importance of each feature can be determined for each channel number. Therefore, for example, with 100 channels, 100 feature importance values ​​can be determined. In the case where multiple defects are reflected in image 1 and there are multiple prediction results, the importance of each feature is determined for each prediction result.

[0078] In step S108, the feature importance calculation unit 14 sorts the feature importance calculated in step S107 in descending order of value, and determines the A (first number) channels that are important for the prediction result (class c, position box_pre). If multiple defects are reflected in the camera image 1 and there are multiple prediction results, the above processing is performed for each prediction result.

[0079] Figure 5 This is an example of the result obtained from the feature importance calculation unit 14. For example... Figure 5 As shown, through steps S105 to S108, the A important channels for the prediction results are obtained.

[0080] use Figure 6 Figure 8 illustrates the method for calculating the statistics stored in the Statistical Information Database (DB) 15.

[0081] Figure 6 This is a block diagram representing the structure of the image recognition system 100. The image recognition system 100, besides... Figure 2 In addition to the structure described in the text, it can also have a feature importance database (DB)20.

[0082] The image recognition unit 2 performs a prediction on the learning image 18 and outputs a prediction result 19. The learning image 18 is the image used by the image recognition unit 2 during learning. The learning image 12 may also use a portion of the image used during learning.

[0083] The feature importance calculation unit 14 calculates the feature importance for the prediction result 19 and saves the result in the feature importance database (DB) 20.

[0084] The statistics calculation unit 21 calculates the statistics based on the results stored in the characteristic quantity importance database (DB) 20, and saves the results to the statistical information database (15). Specific examples of the statistics will be described later.

[0085] Figure 7 This is a flowchart illustrating the process by which the statistics calculation unit 21 calculates the statistics. The following is a description of... Figure 7 The steps are explained below.

[0086] In step S109, the image recognition unit 2 performs prediction on the learning image 18 and outputs the prediction result 19.

[0087] In step S110, the feature importance calculation unit 14 calculates the differential of the feature map of the image recognition unit 2 with respect to the prediction result 19 through error backpropagation, and calculates the importance of the feature map for each channel of the prediction result 19. The calculation process is the same as in S105.

[0088] In step S111, the feature importance calculation unit 14 weights the features held by the feature map using importance and regional information of the prediction results. The calculation process is the same as in S106.

[0089] In step S112, the feature importance calculation unit 14 calculates the feature importance for each channel based on the result of step S111. The calculation process is the same as in S107.

[0090] In step S113, the feature importance calculation unit 14 sorts the feature importance in descending order of the values ​​and determines the B (second number) important channels for prediction.

[0091] In step S114, the feature importance calculation unit 14 saves the results in the feature importance database (DB) 20. At this time, the results are saved according to the level of the prediction results. An example of the results of this step is... Figure 8A The explanation is provided below.

[0092] In step S115, it is determined whether all learning images have been processed. If all learning images have been processed (yes), proceed to step S116. If not all learning images have been processed (no), return to step S109 and repeat the processing after step S109.

[0093] In step S116, the statistics calculation unit 21 calculates statistics based on the results stored in the feature importance database (DB) 20, and determines the statistically significant C channels (the third number) for prediction of the learning image according to the level distinction. For example, based on the results stored in the feature importance database (DB) 20, for each level, the channel numbers of the top C channels that enter Rank B or below most frequently are determined. One example of the result of this step is... Figure 8B The explanation is provided below.

[0094] In step S117, the statistics calculation unit 21 saves the calculated results to the statistics information database (DB) 15 according to the level distinction.

[0095] Figure 8AThis is an example of data stored in feature importance database (DB) 20 in S114. For example... Figure 8A As shown, B important channels derived for each prediction result are stored separately for each level. Figure 8A In this context, each table corresponds to a prediction result (i.e., the result that identifies a defect).

[0096] Figure 8B This is an example of data stored in the statistics database (DB) 15 in S116. For example... Figure 8B As shown, the C important channels obtained from the feature importance database (DB)20 are stored separately according to their level.

[0097] The comparison unit 16 compares the feature importance calculated by the feature importance calculation unit 14 with the statistics related to the feature importance stored in the statistical information database (DB) 15 to determine whether the prediction result is correct. When the prediction level is X, the A important channels for the prediction result calculated by the feature importance calculation unit 14 are compared with the C important channels corresponding to level X stored in the statistical information database (DB) 15. Regarding the determination of whether the prediction result is correct, for example, if N or more of the A important channels calculated by the feature importance calculation unit 14 (threshold) are included in the C important channels stored in the statistical information database (DB) 15, then the prediction is considered correct; otherwise, it is considered incorrect.

[0098] Figure 9 This is a conceptual representation of the determination of the correctness of a prediction result based on a distribution comparison. The comparison unit 16 can also determine the correctness of the prediction result by comparing the distribution of feature importance with the statistical distribution of feature importance. In this case, the feature importance calculation unit 14 calculates the distribution related to the channel number and feature importance for the prediction result. The statistics calculation unit 21 stores the statistical distributions related to the channel number and feature importance of the prediction result for the learning image in the statistical information database (DB) 15 according to different levels. The comparison unit 16 calculates the distance between the distribution calculated by the feature importance calculation unit 14 and the statistical distribution stored in the statistical information database (DB) 15 for the prediction result. If the distance is less than or equal to a threshold D, the prediction result is determined to be correct; otherwise, it is determined to be incorrect. The distance between distributions can be measured, for example, using L1 distance, L2 distance, KL (Kullback-Leibler), etc.

[0099] <Implementation Method 1: Summary>

[0100] The image recognition system 100 of this embodiment 1 uses parameters representing the magnitude of the influence of image features on the recognition result to calculate the importance of each feature for each object shape and for each type of feature. Furthermore, it determines whether the recognition result is correct by comparing the feature importance with its statistical value. Because the comparison of feature importance with its statistical value is performed for each object shape, even when multiple object shapes are reflected in the image, it is possible to determine whether the recognition result is correct for each object shape.

[0101] The image recognition system 100 of this embodiment 1 uses a parameter representing the magnitude of the influence of an image region on the recognition result to calculate the importance of a feature quantity for each object shape and according to each type of feature quantity. Therefore, even when the object shape is scattered throughout the image, it is possible to determine whether the recognition result is correct for each object shape.

[0102] <Implementation Method 2>

[0103] Figure 10 This is a block diagram illustrating the structure of the image recognition system 100 according to Embodiment 2 of the present invention. In this Embodiment 2, the image recognition model is evaluated or compared based on the result of the misidentification determination unit 6.

[0104] Model Database (DB)22 is a database that stores multiple image recognition models that have learned by changing teacher data, learning conditions, etc.

[0105] The model reading unit 23 selects a model from the model database (DB) 22 and reads it into the image recognition unit 2.

[0106] Evaluation image 24 is an image used for evaluating the model and is unlabeled data. These are, for example, appropriate images obtained from images taken by an inspection device.

[0107] The judgment result database (DB) 26 is a database that stores the judgment results of the misidentification judgment unit 6.

[0108] The model evaluation unit 27 evaluates the performance of the model read into the image recognition unit 2 based on the results stored in the judgment result database (DB) 26, and saves the results to the evaluation result database (DB) 28.

[0109] The model comparison unit 29 compares and evaluates the models based on the results stored in the evaluation result database (DB) 28, and saves the results to the comparison result database (DB) 30.

[0110] The model monitoring unit 32 monitors the model based on the results stored in the determination result database (DB) 26, and saves the results in the monitoring database (DB) 33. Details regarding the model monitoring unit 32 and the monitoring database (DB) 33 will be explained in the embodiments described later.

[0111] Display unit 31 is a display device that displays misidentification judgment results, model evaluation results, model comparison results, and model monitoring results. Display unit 31 can also display the various GUIs (Graphical User Interfaces) described later.

[0112] Figure 11 This is a flowchart illustrating the operation of the image recognition system 100 in Embodiment 2. Hereinafter, [the following will describe...] Figure 11 The steps are explained below.

[0113] In step S118, the model reading unit 23 selects a model from the model database (DB) 22 and reads it in. The image recognition unit 2 then acquires the model.

[0114] In step S119, the image recognition unit 2 performs prediction on the evaluation image 24 and outputs the prediction result 25.

[0115] In step S120, the misidentification determination unit 6 determines whether the prediction result 25 is correct and saves it to the determination result database (DB) 26.

[0116] In step S121, it is determined whether all evaluation images have been processed. If all evaluation images have been processed (yes), proceed to step S122. If not all evaluation images have been processed (no), return to step S119 and repeat the processing after step S119.

[0117] In step S122, the model evaluation unit 27 evaluates the performance of the model based on the results stored in the judgment result database (DB) 26, and stores the evaluation results in the evaluation result database (DB) 28 in association with the evaluation model.

[0118] In step S123, it is determined whether all models stored in the model database (DB) 22 have been processed. If all models have been processed (yes), proceed to step S124. If not all models have been processed (no), return to S118.

[0119] In step S124, the model comparison unit 29 compares and evaluates the model based on the results stored in the evaluation result database (DB) 28, and saves the results to the comparison result database (DB) 30.

[0120] Figure 12 This is an example of a GUI that displays the misidentification results stored in the results database (DB) 26 and the model evaluation results stored in the evaluation results database (DB) 28. For example... Figure 12 As shown, the GUI displays (1) evaluation data selection section, (2) model selection section, (3) image selection section, (4) image confirmation section, (5) prediction / judgment result confirmation section, (6) level selection section, (7) evaluation result confirmation section, etc.

[0121] Evaluation data is selected through the (1) evaluation data selection section, and the model is selected through the (2) model selection section.

[0122] In the (5) prediction / determination result confirmation unit, the prediction result of the image recognition unit 2 for the image selected by the (3) image selection unit and displayed by the (4) image confirmation unit is displayed, and the determination result of the misidentification determination unit 6 for the correctness of these prediction results is displayed.

[0123] In the evaluation result confirmation section (7), the evaluation results based on the model evaluation section 27 are displayed. Evaluation indicators include, for example, the total number of identifications, the number of inference misidentifications, the inference misidentification rate, and the inference correct answer rate. The evaluation results for each level can be displayed through the level selection section (6).

[0124] Figure 13 This is an example of a GUI displaying the comparison results stored in the comparison results database (DB) 30. For example... Figure 13 As shown, the GUI displays (1) an evaluation data selection section, (2) a comparison condition setting section, and (3) a comparison result confirmation section.

[0125] Evaluation data is selected through the evaluation data selection section (1).

[0126] In the comparison condition setting section (2), specific settings are made, which are specific settings for comparing and evaluating models. For example, the indicators to be compared, such as model confidence, are set. Model confidence is an indicator that quantitatively represents the "reliability" of the prediction results of an image recognition model. The higher the value, the more likely the prediction results are.

[0127] In the comparison result confirmation section (3), the comparison results for multiple models are displayed under the conditions set by the comparison condition setting section (2). For example, the best model is displayed when the comparison objects are evaluated according to their level.

[0128] <Implementation Method 3>

[0129] Figure 14This is a flowchart illustrating the operation of the image recognition system 100 according to Embodiment 3 of the present invention. In this Embodiment 3, the state of the model is monitored based on the determination result of the misidentification determination unit 6. The structure of the image recognition system 100 is the same as that of Embodiment 2.

[0130] In step S125, it is determined whether the inspection has ended. If the inspection has ended (yes), this flowchart ends. If the inspection has not ended (no), proceed to step S126.

[0131] In step S126, the inspection device 10 captures a photographic image 1 of the sample 9.

[0132] In step S127, the image recognition unit 2 performs prediction on the camera image 1 and outputs the prediction result 25.

[0133] In step S128, the misidentification determination unit 6 determines whether the prediction result 25 is correct and saves the result in the determination result database (DB) 26.

[0134] In step S129, the model monitoring unit 32 sums up the total number of misidentifications at that time point and saves the result in the monitoring database (DB) 33. The model monitoring unit 32 can also calculate the inferred misidentification rate, inferred correct answer rate, etc., based on the total number of recognitions and the total number of misidentifications.

[0135] In step S130, it is determined whether the number of false identifications during the inspection period is above a threshold. If the number of false identifications during the inspection period is above the threshold (Yes), the process proceeds to step S131, and the model monitoring unit 32 issues a warning. If the number of false identifications during the inspection period is not above the threshold (No), the process returns to step S125, and the processing after step S125 is executed again. Alternatively, the determination can be made based on whether the inference false identification rate during the inspection period is above the threshold or the inference correct answer rate during the inspection period is below the threshold.

[0136] Figure 15 This is an example of a GUI output by the display unit 31. Figure 15 This is an example of displaying the results saved in the monitoring database (DB) 33 to the GUI. For example... Figure 15 As shown, the GUI displays (1) a period selection section and (2) a monitoring result confirmation section.

[0137] (2) The monitoring result confirmation unit displays the results stored in the monitoring database (DB) 33. For example... Figure 15 As shown, for example, the horizontal axis represents the inspection period, and the vertical axis represents the inferred false positives and the inferred false positive rate, i.e., the chart. The inspection period on the horizontal axis can be changed by the period selection section in (1). Figure 15In the example, the inference misidentification rate is calculated monthly, and if it exceeds the preset value, the model monitoring unit 32 issues a warning.

[0138] <Regarding variations of the present invention>

[0139] This invention is not limited to the embodiments described above, and includes various modifications. For example, the embodiments described above are those described in detail to aid in understanding the invention, and are not limited to having all the described structures. Furthermore, a portion of the structure of one embodiment can be replaced with the structure of another embodiment, and it is also possible to add structures of other embodiments to the structure of one embodiment. In addition, for a portion of the structure of each embodiment, other structures can be added, deleted, or replaced.

[0140] In the above embodiments, the case where the image recognition unit 2 outputs the type (level) and location of defects as the prediction result has been described. The image recognition system 100 of the present invention can also be applied to the case where the image recognition unit 2 performs level classification (segmentation) on a per-pixel basis of the input image. In this case, for example, the segmented region obtained from the prediction result of the image recognition unit 2 is set as the designated region and set as the input of the misidentification determination unit 6.

[0141] In the above embodiments, the present invention can also be applied to image recognition systems that identify the shape of any shape other than a defect when it appears in the image. That is, the present invention can generally be applied to systems that identify the type and location of objects in an image.

[0142] In the above embodiments, an image (typical image) of the sample to be inspected can also be obtained in advance, and the prediction result of the image recognition unit 2 can be compared with the typical image to determine whether the prediction result is correct.

[0143] In the above embodiments, the feature importance calculation unit 14 can, for example, perform calculations by assigning a level score to a representative pixel in a user-specified region. Figure 4 The processing shown yields the importance of the feature quantities. Representative pixels are determined, for example, by selecting the pixel with the highest score in the input region. The statistics calculation unit 21 calculates the importance of each specified region based on the prediction results of the image used in the learning process of the image recognition unit. Figure 7 The process shown yields statistics related to the importance of the feature. Through the above processing, the misidentification determination unit 6 can determine whether the prediction result is correct for each specified region in the prediction result of the image recognition unit 2, and based on the result, can perform model evaluation, model comparison, and model monitoring.

[0144] In the above embodiments, the coefficient α in mathematical formula 2 can also be omitted, and G can be calculated using only A and S. Incidentally, according to the inventors' experiments, even in this case, the accuracy of the correctness determination is sufficient.

[0145] In the above embodiments, the various functional units (image recognition unit 2, misidentification determination unit 6, statistical calculation unit 21, model reading unit 23, model evaluation unit 27, model comparison unit 29, display unit 31, and model monitoring unit 32) of the image recognition system 100 can also be constructed by hardware such as circuit devices with the function installed, or by executing software with the function installed by a computing device (e.g., a central processing unit).

[0146] In the above embodiments, each database can be configured by storing the recorded data files in a storage device. The database management function for accessing the database can be installed in each functional unit of the image recognition system 100, or a separate database management system can be set up and accessed through it.

[0147] In the above embodiments, the inspection device 10 may be configured as part of the image recognition system 100, or the image recognition system 100 may be configured as a device independent of the inspection device 10.

Claims

1. An image recognition system for recognizing shapes contained in an image, characterized in that, have: The image recognition unit extracts features from an input image based on the results of machine learning, and uses the features to identify the shapes of objects included in the input image; and The misidentification determination unit determines whether the recognition result of the image recognition unit is correct. The misidentification determination unit includes: The feature importance calculation unit calculates the importance of the feature; A statistical information database stores statistics related to the importance level; and The comparison unit determines whether the identification result is correct by comparing the importance and the statistical measure. The feature importance calculation unit calculates the importance of each object shape identified by the image recognition unit and according to each type of feature. The statistical information database stores the statistics according to each type of object shape and according to each type of feature quantity. The comparison unit compares the importance of each type of feature calculated by the feature importance calculation unit with the statistics of each type of feature stored in the statistical information database for each object shape identified by the image recognition unit, thereby determining whether the recognition result is correct according to each object shape identified by the image recognition unit.

2. The image recognition system according to claim 1, wherein, The feature importance calculation unit uses a feature importance parameter, which represents the magnitude of the influence of the feature quantity on the shape of the object recognized by the image recognition unit, to calculate the importance. In addition to using the feature importance parameter, the feature importance calculation unit also uses a region importance parameter, which represents the magnitude of the influence of the image region within the input image when the image recognition unit recognizes the shape of the object, to calculate the importance.

3. The image recognition system according to claim 1, wherein, The feature importance calculation unit uses a feature importance parameter, which represents the magnitude of the influence of the feature quantity on the shape of the object recognized by the image recognition unit, to calculate the importance. The feature importance calculation unit calculates the feature importance parameter using the ratio of the increment of the confidence score of the recognition result to the increment of the feature at each pixel position of the input image.

4. The image recognition system according to claim 1, wherein, The feature importance calculation unit uses a region importance parameter, which represents the magnitude of the influence of image regions within the input image when the image recognition unit recognizes the shape of the object, to calculate the importance. The feature importance calculation unit calculates the region importance parameter by dividing the confidence score of the recognition result by the feature quantity for each image region by the ratio between the maximum value of the differential value in the input image.

5. The image recognition system according to claim 1, wherein, The image recognition system also includes a statistical calculation unit for creating the statistical information database. The statistical calculation unit creates a first list, based on each object shape identified by the image recognition unit, listing a first number of types of the feature quantities in descending order of importance. The statistical calculation unit stores a second list, which lists a second number of the types of the feature quantities in descending order of frequency, as the statistical quantity in each of the first lists according to the object shape identified by the image recognition unit. The feature importance calculation unit creates a third list that enumerates a third number of feature types in descending order of importance for each object shape identified by the image recognition unit. If the number of feature quantities listed in the third list that exceed a certain threshold is included in the second list, the comparison unit determines that the recognition result for the object shape is correct; otherwise, it determines that it is incorrect.

6. The image recognition system according to claim 1, wherein, The image recognition system also includes a statistical calculation unit for creating the statistical information database. The feature importance calculation unit creates a first distribution describing the types of features and the distribution of their importance for each object shape identified by the image recognition unit. The statistical calculation unit creates a second distribution describing the distribution of the types of feature quantities and the importance of each type of object shape identified by the image recognition unit. The comparison unit calculates the distance between the first distribution and the second distribution. If the distance is below a threshold, the comparison unit determines that the recognition result for the object shape is correct; otherwise, it determines that it is incorrect.

7. The image recognition system according to claim 1, wherein, The image recognition system also has the following features: The determination result database stores the recognition results of the image recognition unit or the determination results of the misrecognition determination unit for one or more of the input images; An evaluation results database stores the evaluation results of the performance of the image recognition unit; as well as The model evaluation unit evaluates the performance of the image recognition unit based on the results stored in the judgment result database, and stores the evaluation results in the evaluation result database.

8. The image recognition system according to claim 7, wherein, The image recognition system also has the following features: A model database that stores one or more image recognition models that have been pre-implemented with machine learning to identify shapes contained in images; The model reading unit enables the image recognition unit to read the image recognition model stored in the model database; as well as The model comparison unit evaluates the image recognition model based on one or more evaluation results stored in the evaluation result database. The misidentification determination unit stores the determination result in association with the image recognition model in the determination result database. The model evaluation unit stores the evaluation results in the evaluation result database in association with the image recognition model.

9. The image recognition system according to claim 7, wherein, The image recognition system also includes a model monitoring unit, which determines whether the image recognition unit is performing an abnormal action based on the judgment results stored in the judgment result database, and outputs a warning indicating that the image recognition unit is performing an abnormal action if an abnormal action is being performed.

10. The image recognition system according to claim 1, wherein, The shape of the object was identified by using machine learning through convolutional neural networks.

11. The image recognition system according to claim 1, wherein, The image recognition unit identifies the type and location of objects reflected in the input image based on the object's shape.

12. The image recognition system according to claim 1, wherein, The misidentification determination unit determines the success or failure of the image recognition unit's recognition result by comparing a pre-input typical image with the input image.

Images