Image Recognition System

Abstract

To provide a technique capable of, in an image recognition system that recognizes shapes included in an image, determining a success or a failure of individual prediction results for those shapes even if a plurality of shapes appear in the image.SOLUTION: An image recognition system according to the present invention calculates an importance degree of feature quantities for each target shape recognized in an image and for each type of the feature quantities, and determines a success or a failure of a recognition result by comparing, for each target shape, the importance degree with a statistic amount for the type of the feature quantity.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to an image recognition system that recognizes a shape contained in an image. [Background technology]

[0002] In manufacturing lines for semiconductors and liquid crystal panels, if a defect occurs early in the process, the work in the subsequent processes will be wasted. Therefore, inspection processes are set up at key points in the process to confirm and maintain the desired yield while proceeding with the production. In these inspection processes, for example, critical dimension-SEM (CD-SEM) and defect review SEM (Defect Review SEM), which are applications of scanning electron microscopes (SEM), are used.

[0003] In the inspection process, the presence or absence of defects or abnormalities is confirmed in the images captured by the inspection device. In recent years, highly accurate automatic inspection has become possible thanks to image processing models constructed by machine learning. However, since the characteristics of the sample to be inspected change depending on the manufacturing process, it is necessary to retrain the image processing model at the manufacturing site in order to maintain high inspection accuracy. In such cases, it is necessary to compare and evaluate the performance of the retrained model with the existing model, and to confirm whether the retrained model is functioning normally. In other words, performance evaluation of the image recognition model is necessary. Generally, performance evaluation is performed using labeled data or confirmation by a third party, but it is difficult to perform these at the manufacturing site from the standpoint of cost, time, etc. Therefore, it is necessary to automatically evaluate the performance of the model using unlabeled data.

[0004] Background art in this technical field includes, for example, a technique such as Patent Document 1. Patent Document 1 aims to provide a technique that "can present information on the accuracy of a classification result in a simple manner in an image classification device and image classification method that perform classification based on image features." It describes a technique in which "a defect image X classified into category A by an arbitrary classification algorithm is treated as a calculation target, and the accuracy of the classification result is calculated. For each of a plurality of types of feature quantities V1 to V8, a range of values ​​possessed by a typical image belonging to the classification category is obtained as a typical range. Of the feature quantities representing the calculation target image X, those whose values ​​are within the typical range are voted for, and the ratio of the number of votes to the number of types of feature quantities is output as the accuracy." (see abstract). [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2013-077127 A Summary of the Invention [Problem to be solved by the invention]

[0006] The technology of Patent Document 1 outputs a degree of accuracy that indicates the likelihood of a classification result by a classification algorithm (image recognition model). This makes it possible to determine whether the classification result of the image recognition model is correct or not, so it is believed that it is possible to automatically evaluate the performance of a model using unlabeled data.

[0007] However, conventional image classification devices such as those described in Patent Document 1 calculate feature amounts for an entire input image. In other words, even if multiple defects exist in an image, feature amounts that identify the multiple defects as a whole are calculated. In this case, what is identified by the feature amount is a collection of multiple defects, and each defect is not identified individually. Therefore, it is difficult to judge whether the prediction result for each defect is successful or not.

[0008] The present invention has been made in consideration of the above-mentioned problems, and aims to provide a technology in an image recognition system that recognizes shapes contained in an image, which is capable of determining the success or failure of the individual prediction results for those shapes even when multiple shapes are captured in the image. [Means for solving the problem]

[0009] The image recognition system of the present invention calculates the importance of features for each target shape recognized in an image and for each type of feature, and determines whether the recognition result is correct by comparing the importance with statistics for each type of feature for each target shape. Effect of the Invention

[0010] According to the image recognition system of the present invention, in an image recognition system that recognizes shapes contained in an image, even if multiple shapes are captured in the image, it is possible to determine the success or failure of the prediction results for each of the shapes. Problems, configurations, and effects other than those described above will become apparent from the description of the embodiments below. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 is a diagram conceptually illustrating a configuration of an image recognition system 100 according to a first embodiment. [Diagram 2] FIG. 1 is a block diagram showing a configuration of an image recognition system 100. [Diagram 3] 1 is a flowchart illustrating the operation of the image recognition system 100. [Figure 4] 13 is a flowchart showing the process of a feature amount importance calculation unit 14. [Diagram 5] 13 is an example of a result obtained by the feature importance calculation unit 14. [Figure 6] FIG. 1 is a block diagram showing a configuration of an image recognition system 100. [Figure 7] 4 is a flowchart illustrating a procedure in which the statistics calculation unit 21 calculates statistics. [Figure 8A]This is an example of data to be stored in the feature importance database (DB) 20 in S114. [Figure 8B] This is an example of data to be stored in the statistical information database (DB) 15 in S116. [Figure 9] FIG. 13 is a diagram conceptually illustrating the determination of the accuracy of a prediction result by comparing distributions. [Figure 10] FIG. 1 is a block diagram showing a configuration of an image recognition system 100 according to a second embodiment. [Figure 11] 10 is a flowchart illustrating the operation of the image recognition system 100 according to the second embodiment. [Figure 12] 1 is an example of a GUI that displays the misrecognition judgment results stored in the judgment result database (DB) 26 and the model evaluation results stored in the evaluation result database (DB) 28. [Figure 13] 1 is an example of a GUI that displays the comparison results stored in a comparison result database (DB) 30. [Figure 14] 11 is a flowchart illustrating the operation of the image recognition system 100 according to the third embodiment. [Figure 15] 4 is an example of a GUI output by the display unit 31. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] <Embodiment 1> 1 is a diagram conceptually illustrating a configuration of an image recognition system 100 according to a first embodiment of the present invention. The image recognition system 100 includes an image recognition unit 2, an error recognition determination unit 6, and the like. The image recognition unit 2 obtains a prediction result 3 by making a prediction on a captured image 1. The error recognition determination unit 6 determines whether the prediction result 3 is correct or not.

[0013] As shown in prediction result 3, image recognition unit 2 individually predicts the type and position of the defect shown in captured image 1, as shown in prediction result 4 and prediction result 5. The results predicted by image recognition unit 2 are input to error recognition determination unit 6, which individually determines whether these results are correct or not. In other words, error recognition determination unit 6 determines whether the predicted type of defect is correct for each area of ​​the predicted results, and outputs prediction result 4 as determination result 7, and prediction result 5 as determination result 8.

[0014] Fig. 2 is a block diagram showing the configuration of the image recognition system 100. A specific configuration for realizing the functions described in Fig. 1 will be described with reference to Fig. 2.

[0015] The inspection device 10 captures an image 1 of a sample 9. The sample 9 is, for example, a semiconductor wafer. The inspection device 10 is, for example, a defect inspection device using a mirror electron microscope that forms an image of mirror electrons, an optical defect inspection device, or the like.

[0016] The image recognition unit 2 performs defect inspection on the acquired captured image 1. The image recognition unit 2 extracts features from the captured image 1 and detects defects in the captured image 1 from the extracted features. When multiple defects are present in the captured image 1, the image recognition unit 2 predicts these defects individually. Therefore, the image recognition unit 2 is provided with an image recognition model capable of predicting the type and position of defects. As the image recognition model provided in the image recognition unit 2, for example, a Single Shot Multibox Detector (SSD) or RetinaNet configured with a Convolution Neural Network (CNN) can be used.

[0017] The error recognition determination unit 6 includes a feature importance calculation unit 14, a statistical information database (DB) 15, and a comparison unit 16. The processing content of each component will be described in detail later.

[0018] The feature importance calculation unit 14 receives the prediction result 13 as an input and calculates the feature importance from the prediction result 13. The feature importance represents the importance that the feature extracted by the image recognition unit 2 has with respect to the prediction result. Specific examples of the importance will be described later.

[0019] A statistical information database (DB) 15 stores statistics relating to the feature importance calculated by the feature importance calculation unit 14 .

[0020] The comparison unit 16 compares the feature importance calculated by the feature importance calculation unit 14 with the statistics stored in the statistical information database (DB) 15. If the feature importance calculated by the feature importance calculation unit 14 deviates from the statistics stored in the statistical information database (DB) 15, the comparison unit 16 determines that the prediction result 13 is an error, and outputs it as a determination result 17.

[0021] Fig. 3 is a flowchart illustrating the operation of the image recognition system 100. Each step in Fig. 3 will be described below.

[0022] In step S101, the inspection device 10 captures the image 1 of the sample 9.

[0023] In step S102, the image recognition unit 2 performs image processing on the captured image 1 to predict the type and position of a defect appearing in the captured image 1, and outputs the prediction result 13.

[0024] In step S103, the feature importance calculation unit 14 calculates the feature importance for each prediction result.

[0025] In step S104, the comparison unit 16 compares the feature importance calculated by the feature importance calculation unit 14 with the statistics stored in the statistical information database (DB) 15 to determine whether each prediction result is correct.

[0026] The processing contents of each of the feature importance calculation section 14, the statistical information database (DB) 15, and the comparison section 16 constituting the error recognition determination section 6 will be described in detail with reference to FIGS.

[0027] 4 is a flowchart showing the process of the feature importance calculation unit 14. Each step in FIG. 4 will be described below.

[0028] In step S105, the feature importance calculation unit 14 uses error backpropagation to calculate the derivative of the feature map of the image recognition unit 2 with respect to the prediction result. Using this derivative value, the importance with respect to the prediction result is calculated for each channel (type of feature) of the feature map. The feature map holds the features extracted with respect to the captured image 1. The processing of this step is shown in Equation 1.

[0029]

number

[0030] In number 1, y c,box_pre is the score for class c (type of defect) predicted by the image recognition unit 2, and box_pre represents the predicted position. i,j,k represents the feature map of the image recognition unit 2, i and j represent the vertical and horizontal pixel numbers of the feature map, respectively, and k represents the channel number. u and v represent the number of vertical and horizontal pixels of the feature map, respectively. z is u × v. That is, in step S105, the derivative of the feature map with respect to the score value for class c and position box_pre is calculated, and the average value is calculated for each channel. The α obtained from this k,c,box_pre represents the importance of the feature held in the feature map of channel number k with respect to the prediction result (class is c, position is box_pre). The importance is calculated for each channel, so for example, if there are 100 channels, 100 different importances will be calculated. If captured image 1 contains multiple defects and there are multiple prediction results, the importance is calculated for each prediction result. This importance represents the influence that feature type k has on the recognition result.

[0031] In step S106, the feature importance calculation unit 14 weights the feature held in the feature map by the importance calculated in step S105 and the area information of the prediction result. This process is shown in Equation 2. k,c,box_pre is the region information of the prediction result and is calculated by Equation 3. k,c,box_pre represents the importance of the region for the prediction result (class is c, position is box_pre) for each pixel of the feature map of channel number k. The region information of the prediction result may be a mask in which the inside of the region of the prediction result is set to 1 and the other regions to 0, or a template region in which an important region is set in advance. If the captured image 1 contains multiple defects and there are multiple prediction results, the process of equation 2 is carried out for each prediction result. In step S106, the feature importance calculation unit 14 may weight the features held in the feature map only by the importance calculated in step S105. G k,c,box_pre or S k,c,box_pre represents the degree of influence that an image region has on the recognition result.

[0032]

number

[0033]

number

[0034] In step S107, the feature importance calculation unit 14 calculates the G k,c,box_pre The feature importance is calculated for each channel number from k,c,box_pre In this case, the feature importance can be calculated by, for example, calculating the average of values ​​only in the area near the predicted result, or calculating the average of only values ​​equal to or greater than a preset threshold. The above process makes it possible to calculate the feature importance for each channel number, so for example, if there are 100 channels, 100 different feature importances can be calculated. If the captured image 1 contains multiple defects and there are multiple predicted results, the feature importance is calculated for each predicted result.

[0035] In step S108, the feature importance calculation unit 14 sorts the feature importance calculated in step S107 in descending order of value, and determines A important channels (first number) for the prediction result (class is c, position is box_pre). If multiple defects are captured in the captured image 1 and there are multiple prediction results, the above process is performed for each prediction result.

[0036] Fig. 5 is an example of the result obtained by the feature importance calculation unit 14. As shown in Fig. 5, A channels that are important to the prediction result are obtained by steps S105 to S108.

[0037] A method for determining the statistics stored in the statistical information database (DB) 15 will be described with reference to FIGS.

[0038] 6 is a block diagram showing the configuration of the image recognition system 100. The image recognition system 100 may include a feature importance database (DB) 20 in addition to the configuration described in FIG.

[0039] The image recognition unit 2 performs prediction on the training images 18 and outputs the prediction results 19. The training images 18 are images used during training of the image recognition unit 2. As the training images 12, some of the images used for training may be used.

[0040] The feature importance calculation unit 14 calculates the feature importance for the prediction result 19 and stores the result in a feature importance database (DB) 20 .

[0041] A statistics calculation unit 21 calculates statistics from the results stored in the feature importance database (DB) 20, and stores the results in the statistical information database (15). Specific examples of statistics will be described later.

[0042] 7 is a flowchart illustrating the procedure for calculating statistics by the statistics calculation unit 21. Each step in FIG. 7 will be described below.

[0043] In step S109, the image recognition unit 2 performs prediction on the training image 18 and outputs the prediction result 19.

[0044] In step S110, the feature importance calculation unit 14 calculates the derivative of the feature map of the image recognition unit 2 with respect to the prediction result 19 by error backpropagation, and obtains the importance with respect to the prediction result 19 for each channel of the feature map. The calculation procedure is the same as in S105.

[0045] In step S111, the feature importance calculation unit 14 weights the feature held in the feature map by the importance and the area information of the prediction result. The calculation procedure is the same as in S106.

[0046] In step S112, the feature importance calculation unit 14 calculates the feature importance for each channel from the result of step S111. The calculation procedure is the same as in S107.

[0047] In step S113, the feature importance calculation unit 14 sorts the feature importance in descending order of value, and determines B channels (a second number) that are important for prediction.

[0048] In step S114, the feature importance calculation unit 14 stores the results in the feature importance database (DB) 20. At this time, the results are stored by class of the prediction results. An example of the results of this step will be described with reference to FIG. 8A.

[0049] In step S115, it is determined whether or not the process has been performed on all the training images. If the process has been performed on all the training images (YES), the process proceeds to step S116. If the process has not been performed on all the training images (NO), the process returns to step S109 and the processes from step S109 onwards are executed again.

[0050] In step S116, the statistics calculation unit 21 calculates statistics from the results stored in the feature importance database (DB) 20, and determines C statistically determined important channels (a third number) for the prediction of the training image for each class. This is done, for example, by determining the top C channel numbers that have been included in Rank B most frequently for each class from the results stored in the feature importance database (DB) 20. An example of the result of this step will be described in FIG. 8B.

[0051] In step S117, the statistics calculation unit 21 stores the obtained results in the statistical information database (DB) 15 for each class.

[0052] Fig. 8A is an example of data stored in the feature importance database (DB) 20 in S114. As shown in Fig. 8A, B important channels obtained for each prediction result are stored by class. In Fig. 8A, one table corresponds to one prediction result (i.e., the result of identifying one defect).

[0053] Fig. 8B is an example of data stored in S116 in the statistical information database (DB) 15. As shown in Fig. 8B, C important channels obtained from the feature importance database (DB) 20 are stored by class.

[0054] The comparison unit 16 compares the feature importance calculated by the feature importance calculation unit 14 with statistics related to the feature importance stored in the statistical information database (DB) 15 to determine whether the prediction result is correct. When the prediction class is X, the comparison unit 16 compares the A important channels for the prediction result calculated by the feature importance calculation unit 14 with the C important channels corresponding to class X stored in the statistical information database (DB) 15. The prediction result is determined to be correct, for example, if N or more (threshold) of the A important channels calculated by the feature importance calculation unit 14 are included in the C important channels stored in the statistical information database (DB) 15, and if not, the prediction result is determined to be incorrect.

[0055] FIG. 9 is a diagram conceptually illustrating the judgment of the correctness of the prediction result by comparing the distributions. The comparison unit 16 may compare the distribution of feature importance with the statistical distribution of feature importance to judge the correctness of the prediction result. In this case, the feature importance calculation unit 14 obtains a distribution of channel numbers and feature importance for the prediction result. The statistics calculation unit 21 stores the statistical distribution of channel numbers and feature importance for the prediction result of the learning image in the statistical information database (DB) 15 by class. The comparison unit 16 calculates the distance between the distribution obtained by the feature importance calculation unit 14 for the prediction result and the statistical distribution stored in the statistical information database (DB) 15, and judges the prediction result as correct if the distance is equal to or smaller than a threshold D, or as incorrect if not. The distance between the distributions is measured by, for example, the L1 distance, the L2 distance, or KL (Kullback-Leibler).

[0056] <First embodiment: Summary> The image recognition system 100 according to the first embodiment calculates feature importance for each target shape and for each type of feature using a parameter that indicates the degree of influence that an image feature has on the recognition result. Furthermore, the system judges whether the recognition result is correct or not by comparing the feature importance with its statistics. Since the system compares the feature importance with its statistics for each target shape, even if multiple target shapes are captured in an image, the system can judge whether the recognition result is correct or not for each target shape.

[0057] The image recognition system 100 according to the first embodiment calculates the feature importance for each target shape and for each feature type using a parameter that indicates the degree of influence that an image region has on the recognition result. This makes it possible to determine whether the recognition result is correct for each target shape, even if the target shapes are distributed throughout the image.

[0058] <Embodiment 2> 10 is a block diagram showing a configuration of an image recognition system 100 according to a second embodiment of the present invention. In the second embodiment, based on the result of the recognition error determination unit 6, image recognition models are evaluated or compared.

[0059] A model database (DB) 22 is a database in which a plurality of image recognition models trained by changing the teaching data, the learning conditions, and the like are stored.

[0060] The model reading unit 23 selects a model from a model database (DB) 22 and reads it into the image recognition unit 2 .

[0061] The evaluation images 24 are images for evaluating the model and are unlabeled data. For example, these are appropriate images collected from images captured by an inspection device.

[0062] The determination result database (DB) 26 is a database that stores the determination results by the error recognition determination unit 6 .

[0063] The model evaluation unit 27 evaluates the performance of the model read into the image recognition unit 2 from the results stored in the judgment result database (DB) 26, and stores the evaluation result in an evaluation result database (DB) .

[0064] A model comparison unit 29 compares and evaluates models based on the results stored in an evaluation result database (DB) 28 , and stores the results in a comparison result database (DB) 30 .

[0065] The model monitoring unit 32 monitors the model based on the results stored in the judgment result database (DB) 26, and stores the results in a monitoring database (DB) 33. Details of the model monitoring unit 32 and the monitoring database (DB) 33 will be described in the embodiments described later.

[0066] The display unit 31 is a display device that displays the result of the misrecognition determination, the result of the model evaluation, the result of the model comparison, and the result of the model monitoring. The display unit 31 can also display each GUI (Graphical User Interface) to be described later.

[0067] Fig. 11 is a flowchart for explaining the operation of the image recognition system 100 in the present embodiment 2. Each step in Fig. 11 will be explained below.

[0068] In step S118, the model reading unit 23 selects and reads a model from the model database (DB) 22. The image recognition unit 2 acquires the model.

[0069] In step S119, the image recognition unit 2 performs prediction on the evaluation image 24 and outputs the prediction result 25.

[0070] In step S120, the error recognition determination unit 6 determines whether the prediction result 25 is correct or not, and stores the result in a determination result database (DB) .

[0071] In step S121, it is determined whether or not the processing has been performed on all the evaluation images. If the processing has been performed on all the evaluation images (YES), the process proceeds to step S122. If the processing has not been performed on all the evaluation images (NO), the process returns to step S119 and the processing from step S119 onwards is executed again.

[0072] In step S122, the model evaluation unit 27 evaluates the performance of the model from the results stored in the judgment result database (DB) 26, and stores the evaluation result in the evaluation result database (DB) 28 in association with the evaluation model.

[0073] In step S123, it is determined whether or not the process has been performed for all models stored in the model database (DB) 22. If the process has been performed for all models (YES), the process proceeds to step S124. If the process has not been performed for all models (NO), the process returns to S118.

[0074] In step S 124 , the model comparison unit 29 compares and evaluates the models based on the results stored in the evaluation result database (DB) 28 , and stores the results in the comparison result database (DB) 30 .

[0075] Fig. 12 is an example of a GUI that displays the misrecognition judgment results stored in the judgment result database (DB) 26 and the model evaluation results stored in the evaluation result database (DB) 28. As shown in Fig. 12, the GUI displays (1) an evaluation data selection unit, (2) a model selection unit, (3) an image selection unit, (4) an image confirmation unit, (5) a prediction / judgment result confirmation unit, (6) a class selection unit, (7) an evaluation result confirmation unit, and the like.

[0076] (1) Evaluation data is selected by an evaluation data selection unit, and (2) a model is selected by a model selection unit.

[0077] The (5) prediction / judgment result confirmation unit displays the prediction results of the image recognition unit 2 for the image selected by the (3) image selection unit and displayed in the (4) image confirmation unit, and the judgment results of the misrecognition judgment unit 6 as to whether the prediction results are correct or incorrect.

[0078] (7) The evaluation result confirmation unit displays the evaluation results by the model evaluation unit 27. The evaluation indexes are, for example, the total number of recognitions, the estimated number of misrecognitions, the estimated misrecognition rate, the estimated correct answer rate, etc. (6) The class selection unit can display the evaluation results for each class.

[0079] Fig. 13 is an example of a GUI that displays the comparison results stored in the comparison result database (DB) 30. As shown in Fig. 13, the GUI displays (1) an evaluation data selection section, (2) a comparison condition setting section, (3) a comparison result confirmation section, etc.

[0080] (1) Evaluation data is selected by the evaluation data selection unit.

[0081] (2) The comparison condition setting section performs specific settings for the comparative evaluation of models. For example, the indices to be compared and the model confidence level are set. The model confidence level is an index that quantitatively indicates the "likelihood" of the prediction results of the image recognition model. The higher the value, the higher the probability of the prediction results.

[0082] The comparison result confirmation section (3) displays the comparison results for multiple models based on the conditions set in the comparison condition setting section (2). For example, the optimal model when the comparison targets are evaluated by class is displayed.

[0083] <Embodiment 3> 14 is a flowchart for explaining the operation of the image recognition system 100 according to the third embodiment of the present invention. In the third embodiment, the state of the model is monitored based on the determination result of the error recognition determination unit 6. The configuration of the image recognition system 100 is the same as that of the second embodiment.

[0084] In step S125, it is determined whether the test has ended. If the test has ended (YES), this flow chart ends. If the test has not ended (NO), the process proceeds to step S126.

[0085] In step S126, the inspection device 10 captures the captured image 1 of the sample 9.

[0086] In step S127, the image recognition unit 2 performs prediction on the captured image 1 and outputs the prediction result 25.

[0087] In step S128, the error recognition determination unit 6 determines whether the prediction result 25 is correct or not, and stores the result in the determination result database (DB) .

[0088] In step S129, the model monitoring unit 32 tallies up the total number of recognition errors at that time point, and stores the result in the monitoring database (DB) 33. The model monitoring unit 32 may obtain an estimated recognition error rate, an estimated correct answer rate, etc. from the total number of recognitions and the total number of recognition errors.

[0089] In step S130, it is determined whether the number of erroneous recognitions within the inspection period is equal to or greater than a threshold. If the number of erroneous recognitions within the inspection period is equal to or greater than the threshold (YES), the process proceeds to step S131, where the model monitoring unit 32 issues a warning. If the number of erroneous recognitions within the inspection period is not equal to or greater than the threshold (NO), the process returns to step S125, and the processes from step S125 onwards are executed again. It may also be determined whether the estimated erroneous recognition rate within the inspection period is equal to or greater than a threshold, the estimated correct answer rate within the inspection period is equal to or less than a threshold, etc.

[0090] Fig. 15 is an example of a GUI output by the display unit 31. Fig. 15 is an example of the GUI displaying results stored in the monitoring database (DB) 33. As shown in Fig. 15, the GUI displays (1) a period selection section, (2) a monitoring result confirmation section, etc.

[0091] (2) The monitoring result confirmation unit displays the results stored in the monitoring database (DB) 33. As shown in Fig. 15, for example, a graph is displayed in which the horizontal axis represents the inspection period and the vertical axis represents the estimated number of recognition errors and the estimated recognition error rate. The inspection period on the horizontal axis can be changed in the (1) period selection unit. In the example of Fig. 15, the estimated recognition error rate and the estimated recognition error rate are tallied every month, and if they exceed a preset value, the model monitoring unit 32 issues a warning.

[0092] <Modifications of the present invention> The present invention is not limited to the above-described embodiment, and includes various modified examples. For example, the above-described embodiment has been described in detail to aid in understanding the present invention, and is not necessarily limited to those including all of the configurations described. In addition, it is possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. In addition, it is possible to add, delete, or replace a part of the configuration of each embodiment with another configuration.

[0093] In the above embodiment, it has been described that the image recognition unit 2 outputs the type (class) and position of the defect as a prediction result. The image recognition system 100 of the present invention is also applicable to a case where the image recognition unit 2 performs class classification (segmentation) for each pixel of the input image. In this case, for example, the segment area obtained as a prediction result of the image recognition unit 2 is input to the error recognition determination unit 6 as a designated area.

[0094] In the above embodiment, the present invention can also be applied to an image recognition system that recognizes the shape of any shape other than a defect when the shape appears in the image. In other words, the present invention can be applied to a general system that recognizes the type and position of an object in an image.

[0095] In the above embodiment, an image (typical image) of the sample to be inspected may be acquired in advance, and the prediction result of the image recognition unit 2 may be compared with the typical image to determine whether the prediction result is correct or not.

[0096] In the above embodiment, the feature importance calculation unit 14 can calculate feature importance by, for example, performing the process shown in Fig. 4 on the class score of the representative pixel in the area specified by the user. The representative pixel is determined, for example, by selecting the pixel with the highest class score in the input area. The statistics calculation unit 21 calculates statistics related to feature importance for each specified area of ​​the prediction result for the image used for learning of the image recognition unit, according to the flow shown in Fig. 7. By the above process, the misrecognition determination unit 6 can determine whether the prediction result is correct or not for each specified area in the prediction result of the image recognition unit 2, and model evaluation, model comparison, and model monitoring are possible based on the result.

[0097] In the above embodiment, the coefficient α in Equation 2 may be omitted, and G may be calculated only from A and S. It should be noted that, according to the inventor's experiments, even in this case, the accuracy of the correct / incorrect judgment was sufficient.

[0098] In the above embodiments, each functional unit (image recognition unit 2, misrecognition determination unit 6, statistical quantity calculation unit 21, model reading unit 23, model evaluation unit 27, model comparison unit 29, display unit 31, model monitoring unit 32) of the image recognition system 100 can be configured by hardware such as a circuit device that implements that function, or can be configured by a calculation device (e.g., a Central Processing Unit) executing software that implements that function.

[0099] In the above embodiment, each database can be configured by storing a data file that records records in a storage device. A database management function for accessing a database may be implemented by each functional unit of the image recognition system 100, or a database management system may be provided separately to access the records.

[0100] In the above embodiment, the inspection device 10 may be configured as a part of the image recognition system 100, or the image recognition system 100 may be configured as a device independent of the inspection device 10. [Explanation of symbols]

[0101] 100: Image recognition system 2: Image recognition section 6: Misrecognition determination section 9: Sample 10: Inspection equipment 14: Feature importance calculation unit 15:Statistical information database (DB) 16: Comparison section 20: Feature Importance Database (DB) 21: Statistics calculation part 22: Model database (DB) 23: Model loading section 26: Judgment result database (DB) 27: Model evaluation section 28: Evaluation result database (DB) 29: Model comparison section 30: Comparison result database (DB) 31: Display section 32: Model Monitoring Department 33: Monitoring database (DB)

Claims

1. An image recognition system for recognizing a shape included in an image, an image recognition unit that extracts features from an input image based on a result of performing machine learning, and recognizes a target shape included in the input image using the features; an error recognition determination unit that determines whether a recognition result by the image recognition unit is correct; Equipped with The misrecognition determination unit a feature importance calculation unit for calculating the importance of the feature; a statistical information database for storing statistics relating to said importance; a comparison unit that judges whether the recognition result is correct or not by comparing the importance with the statistics; Equipped with the feature amount importance calculation unit calculates the importance for each of the target shapes recognized by the image recognition unit and for each type of the feature amount; the statistical information database stores the statistics for each type of the target shape and for each type of the feature amount, The comparison unit compares, for each of the target shapes recognized by the image recognition unit, the importance for each of the feature types calculated by the feature importance calculation unit with the statistics for each of the feature types stored in the statistical information database, to determine whether the recognition result is correct for each of the target shapes recognized by the image recognition unit.

1. An image recognition system comprising:

2. the feature amount importance calculation unit calculates the importance using a feature amount importance parameter that indicates a magnitude of an influence of the feature amount when the image recognition unit recognizes the target shape; The feature importance calculation unit calculates the importance by using, in addition to the feature importance parameter, a region importance parameter that indicates the degree of influence of an image region in the input image when the image recognition unit recognizes the target shape.

2. The image recognition system according to claim 1.

3. the feature amount importance calculation unit calculates the importance using a feature amount importance parameter that indicates a magnitude of an influence of the feature amount when the image recognition unit recognizes the target shape; The feature importance calculation unit calculates the feature importance parameter by using a ratio of an increment of the confidence score of the recognition result to an increment of the feature for each pixel position of the input image.

2. The image recognition system according to claim 1.

4. the feature amount importance calculation unit calculates the importance using a region importance parameter representing a magnitude of influence of an image region in the input image when the image recognition unit recognizes the target shape; The feature importance calculation unit calculates the region importance parameter by calculating, for each image region, a ratio between a differential value obtained by differentiating the certainty score of the recognition result by the feature and a maximum value of the differential value in the input image.

2. The image recognition system according to claim 1.

5. The image recognition system further includes a statistics calculation unit that creates the statistical information database, the statistics calculation unit creates a first list in which the types of the feature amounts are listed in a first order of importance for each of the target shapes recognized by the image recognition unit; the statistics calculation unit stores a second list in which a second number of types of the feature amounts are listed in order of frequency of inclusion in each of the first lists in the statistical information database as the statistics for each of the target shapes recognized by the image recognition unit; the feature importance calculation unit creates a third list in which the types of the feature amounts are listed in order of importance, for each of the target shapes recognized by the image recognition unit; The comparison unit determines that the recognition result for the target shape is correct if the second list contains a threshold number or more of the types of the feature amounts listed in the third list, and determines that the recognition result is incorrect if not.

2. The image recognition system according to claim 1.

6. The image recognition system further includes a statistics calculation unit that creates the statistical information database, the feature importance calculation unit creates a first distribution describing a distribution of the types of the feature amounts and the importance levels for each of the target shapes recognized by the image recognition unit; the statistics calculation unit creates a second distribution describing a distribution of the type of the feature amount and the importance for each type of the target shape recognized by the image recognition unit; The comparison unit calculates a distance between the first distribution and the second distribution, If the distance is equal to or smaller than a threshold, the comparison unit determines that the recognition result for the target shape is correct, and if not, determines that the recognition result is incorrect.

2. The image recognition system according to claim 1.

7. The image recognition system further comprises: a determination result database for storing the determination results made by the misrecognition determination unit for one or more of the input images; an evaluation result database for storing the results of evaluating the performance of the image recognition unit; a model evaluation unit that evaluates the performance of the image recognition unit based on the results stored in the judgment result database and stores the evaluation results in the evaluation result database; Equipped 2. The image recognition system according to claim 1.

8. The image recognition system further comprises: A model database that stores image recognition models that have undergone machine learning to recognize shapes contained in images; a model reading unit that reads the image recognition model stored in the model database into the image recognition unit; a model comparison unit that evaluates the image recognition model based on the evaluation results stored in the evaluation result database; Equipped with the misrecognition determination unit stores the determination result in the determination result database in association with the image recognition model; The model evaluation unit stores the evaluation result in the evaluation result database in association with the image recognition model.

8. The image recognition system according to claim 7.

9. The image recognition system further includes a model monitoring unit that determines whether the image recognition unit is operating abnormally based on the judgment result stored in the judgment result database, and outputs a warning to that effect if the image recognition unit is operating abnormally.

8. The image recognition system according to claim 7.

10. The image recognition unit recognizes the target shape by machine learning using a convolutional neural network.

2. The image recognition system according to claim 1.

11. The image recognition unit recognizes the type and position of an object shown in the input image based on the target shape.

2. The image recognition system according to claim 1.

12. The misrecognition determination unit determines whether the prediction result by the image recognition unit is correct by comparing a typical image input in advance with the prediction result by the image recognition unit.

2. The image recognition system according to claim 1.

Images