Method of and apparatus for monitoring and evaluating image classification model

Abstract

To provide a method of and an apparatus for monitoring and evaluating a calibrated image classification model especially for the purpose of discriminating an error, e.g., for use in automatic optical inspection in the field of manufacturing a component.SOLUTION: A method according to the present invention includes: classifying an image with using a calibrated image classification model; determining a value matched with a minimum class dependency as to each of the images classified with use of the calibrated image classification model, based on a softmax distribution predicted for each of the images and a characteristic softmax distribution of a class of the calibrated image classification model; and generating a visual output on a man-machine interface which enables a user to analyze performance of the calibrated image classification model.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] Prior art The present disclosure relates to a computer-implemented method and apparatus for monitoring and evaluating image classification models. Further embodiments relate to the use of the method and to a manufacturing facility including such an apparatus. [Background technology]

[0002] In the field of automated optical inspection (AOI), for example in the field of manufacturing, it is sought to monitor the proper operation of the image classification model being used and evaluate the model's performance. For monitoring and performance evaluation, a data set with class labels is generally required. However, labeled data is generally not available at the time of using the image classification model. Therefore, monitoring is a challenge. Summary of the Invention [Problem to be solved by the invention]

[0003] A problem arises, for example, when a model is used to classify images that differ from, or have little or no resemblance to, the images on which the model was trained. Classifying images that differ from the images on which the model was trained is called Out-of-Distribution Detection (OOD). Known methods for identifying OOD images require access to the intermediate layers of the model and are therefore cumbersome to implement and not intuitive to use in evaluation. [Means for solving the problem]

[0004] Disclosure of the Invention To solve the above problem, a computer-implemented method according to claim 1 and an apparatus according to claim 9 are proposed.

[0005] One embodiment relates to a computer-implemented method for monitoring and evaluating a calibrated image classification model, the method comprising: classifying images using a calibrated image classification model that includes predicting a softmax distribution; determining, for each image classified using the calibrated image classification model, a minimum class-dependent agreement value based on the predicted softmax distribution for the respective image and a characteristic softmax distribution of the class of the calibrated image classification model; and generating a visual output on a human-machine interface that allows a user to analyze the performance of the calibrated image classification model, the visual output including a class-specific display for each class of the calibrated image classification model, wherein in each class-specific display, each image classified into each class using the calibrated image classification model is displayed as one data point, and each data point is plotted as the softmax value of the image for each class across the corresponding class-dependent agreement value, and each data point is displayed in color, the color being selected based on the class of the minimum class-dependent agreement value for the respective image.

[0006] According to the present method, it is assumed that the monitoring and evaluation are performed based on the predicted softmax distribution, i.e., based on the softmax output, i.e., based on the class probabilities predicted by the classifier of the calibrated image classification model.

[0007] For each image, a minimum class-dependent agreement value is determined based on the predicted softmax distribution and the characteristic softmax distribution of the class of the calibrated image classification model, the minimum class-dependent agreement value representing the class that has the smallest deviation from the characteristic softmax distribution of the class of the calibrated image classification model according to the predicted softmax distribution.

[0008] It is further assumed that a visual output is generated on the human-machine interface that allows a user to analyze the performance of the calibrated image classification model. To this end, the visual output includes a class representation for each class of the calibrated image classification model. In each class representation, each image classified into each class using the calibrated image classification model is displayed as one data point, and therefore each class representation includes multiple data points, i.e., the number of data points equal to the number of images classified into each class using the calibrated image classification model. The classification into each class k is performed based on the predicted softmax value p, where

number

[0009] For example, in the first class class display, images classified into the first class based on the predicted softmax distribution using the calibrated image classification model are displayed as data points, i.e., the first class has the highest softmax value in the predicted softmax distribution of the image, and each data point is plotted in the diagram as the softmax value of the first class across the corresponding class-dependent agreement value, i.e., across the agreement value for the first class.

[0010] The visual output provides information about how high the corresponding class-dependent agreement value is in each softmax value of the underlying image for each data point in the class-specific representation for each class. The lower the class-dependent agreement value, the more robust the classification of each represented class works.

[0011] Each data point is displayed in a color. If the image underlying the data point has the smallest class-dependent match value for a first class, the data point is displayed in a first color suggesting the first class. If the image underlying the data point has the smallest class-dependent match value for a second class, the data point is displayed in a second color suggesting the second class. In this manner, additional classes and additional colors are added accordingly.

[0012] Thus, the visual output provides information about when, in the class-specific representation of each class, the highest softmax score of the image underlying the data point corresponds to the class of the class-specific representation, but the softmax distribution is closer to another class, i.e., the class for the image underlying the data point has the smallest class-dependent agreement value. This information can suggest that the image underlying the respective data point is difficult or impossible to classify unambiguously by the model.

[0013] Furthermore, according to the above display in the visual output, the data points underlying the images that are difficult to distinguish by the model are arranged linearly in the diagram with a kind of bend, and a color change occurs at the bend, which can suggest that the images underlying each data point of the class are difficult to distinguish by the model according to the color display of the data point.

[0014] The display of the data points in the class-by-class representation of the visual output in the diagram and the resulting arrangement and color representation allow the user to directly obtain information about the internal conditions prevailing in the technical system, i.e., about the functioning and performance of the calibrated image classification model, based on the displayed information. Starting from the display and the resulting information about the internal conditions, the user can take corresponding measures and / or interact with the calibrated image classification model, for example, to improve the functioning and performance of the model and to avoid malfunctions.

[0015] In the case of a large number of classes, it can be assumed that, for ease of viewing in particular, not all classes are displayed in relation to the minimum class dependency agreement value in the class-by-class display. For example, the number of data points per minimum class dependency agreement value can be determined, and then only classes having a frequency above a predetermined threshold, for example, a frequency above 1% of the number of specific data points, are displayed. Alternatively, multiple separate visualizations of the class-by-class display can be created, in which a subset of classes is displayed in relation to the minimum class dependency agreement value.

[0016] According to one embodiment, it is envisioned that, based on the selection of a data point in each class-based representation of the visual output, the image underlying the data point is displayed in the visual output on the human-machine interface. That is, by selecting a data point, the image underlying the data point is displayed. This selection may be performed, for example, by a corresponding user input via a suitable input unit of the human-machine interface. Suitable input units are, for example, input devices such as a computer mouse, keyboard, trackpad, and / or touchscreen. Selecting a data point may include, for example, moving or holding a mouse pointer or finger over the data point in the sense of a mouse-over effect. In this way, the image underlying each data point becomes intuitively and quickly accessible to the user. This allows the user to quickly and efficiently gain insight into which specific image is underlying each data point.

[0017] The display of data points in the diagram in the class-by-class representation of the visual output, and the resulting arrangement and color display, allow for a clear and intuitive display of the data points, allowing a user to intuitively select which specific image they want to display by selecting the corresponding data point. The resulting arrangement and color display of data points in the diagram are suitable for suggesting which images in the model cause or may cause difficulties and should therefore be displayed to gain insight. That is, the display of data points conveys information to the user that enables the user to efficiently perform interactive exploration and selection of images.

[0018] Thus, the user can gain insight into which specific images the above-mentioned difficulties arise in classification. For example, the user can display desired images by sequentially selecting data points, thereby quickly and easily accessibly discovering similarities and / or differences between the displayed images. Thus, the user can personally investigate the images the model is observing during use, and thus gain a better understanding of the difficulties and / or problems of the image classification model, particularly those that could not be predicted at the time of training.

[0019] According to one embodiment, at least one evaluation of the calibrated image classification model is derived based on the visual output, and the evaluation includes identifying at least one class for which the classification provided by the calibrated image classification model is to be checked and / or identifying at least one data point, in particular an image underlying the data point, for which the classification provided by the calibrated image classification model is to be checked. A classification that is to be checked is understood to be, for example, an insufficient or questionable classification. An insufficient or questionable classification is understood to be, for example, a case in which difficulties arise in classification in relation to that class. For example, two or more classes that are impossible or difficult to distinguish unambiguously by the model in classification can also be identified.

[0020] For example, a relatively high class-dependent agreement value may indicate that classification difficulties occur in relation to that class, and a relatively low class-dependent agreement value may indicate that classification difficulties do not occur in relation to that class.

[0021] For example, a bend with a color change in the diagram may suggest that the model is unable or has difficulty in distinguishing these classes unambiguously according to the color change when classifying them. A bend with a color change occurs in the placement of the class-specific representation for the data points in the diagram, where the highest softmax score for the underlying image of the data points corresponds to the class of the respective class-specific representation, but the softmax distribution is located closer to the other class, i.e., the class for the underlying image of the data points has the smallest class-dependent agreement value.

[0022] According to one embodiment, based on the evaluation of the calibrated image classification model: a) training or retraining the calibrated image classification model with a modified training data set; and / or b) modifying the class definitions of the classes for which the classification to be checked is provided by at least the calibrated image classification model; It is assumed that at least one of the following is performed. For example, the modified training data set can be improved by adding or omitting images from the previously used training data set. In particular, the modified training data set can include images of classes that the model struggles with, or a larger number of images. The class definitions are defined, for example, by a domain expert based on errors to be identified in images, for example, in the field of AOI. Such class definitions can often be somewhat arbitrary. Therefore, modifying the class definitions improves the performance of the model.

[0023] According to one embodiment, determining the minimum class-dependent match value for each image is performed by comparing the predicted softmax distribution of each image with the characteristic softmax distribution of the class of the image classification model based on the Kullback-Leibler divergence. Therefore, using the Kullback-Leibler divergence is advantageous because it requires only the model prediction for the determination, not information from intermediate layers of the model that is typically unavailable during model use. The Kullback-Leibler divergence is described, for example, in arXiv:1911.11132 as the KL matching score and is used to identify images with softmax distributions that differ from those of images typically seen during training. In this context, the KL matching score is used to determine the OOD characteristics of an image.

[0024] Instead of the Kullback-Leibler divergence, other OOD values ​​available on the image plane may be used, for example, the minimum class-dependent matching value for each image can be determined based on the cosine similarity of intra-class feature means (arXiv:2306.14920) or the activation histogram distance (arXiv:2309.04837).

[0025] According to one embodiment, it is assumed that the characteristic softmax distribution of the image classification model is the average softmax distribution of the classes of the validation data set and / or the training data set of the calibrated image classification model. In this case, it may prove advantageous if the method comprises the step of determining the characteristic softmax distribution, i.e. in the form of the average softmax distribution of the classes of the validation data set of the calibrated image classification model.

[0026] According to a further embodiment, it can be assumed that the characteristic softmax distribution of the calibrated image classification model is determined based on the softmax distribution recorded during the development of the model, i.e., the image distribution at the time of development is detected. This may be advantageous, for example, when a validation set and / or a training data set is not available.

[0027] According to one embodiment, it is envisioned that the method for monitoring and evaluating a calibrated image classification model is implemented to classify images in real time during the use of the calibrated image classification model.

[0028] A further embodiment relates to an apparatus for monitoring and evaluating a calibrated image classification model, the apparatus including a computing device configured to execute machine-readable instructions that, when executed by the computing device, are capable of performing the method described above, the apparatus including a human-machine interface, and wherein performing the method according to the embodiment causes a visual output on the human-machine interface, the visual output including a class-specific representation for each class of the calibrated image classification model, wherein each image classified into each class using the calibrated image classification model can be displayed as a data point, and each data point can be plotted as the highest softmax value of the images for each class over a corresponding class-dependent agreement value, and each data point can be displayed in color, and the color can be selected based on the class with the lowest class-dependent agreement value for each image.

[0029] According to one embodiment, it is envisaged that based on the selection of a data point in the respective class-specific representation of the visual output, the image underlying that data point can be displayed in the visual output on the human-machine interface.

[0030] A further embodiment relates to the use of a method for monitoring a calibrated image classification model for an automated optical inspection AOI, for example a calibrated image classification model used for automated optical inspection of components in their manufacture, wherein the calibrated image classification model is a multi-class model, and the calibrated image classification model has been trained to classify errors according to error type, the error types being associated with respective classes of the calibrated image classification model.

[0031] In the field of manufacturing, it is known to perform optical inspection of manufactured components to ensure the quality of the manufacturing process. This allows for identifying undesired damage, such as scratches, impurities, breaks, deformations, discoloration, or the like. In particular, to improve efficiency and reduce costs, optical inspection can be performed automatically, i.e., automated optical inspection AOI can be performed. For example, a calibrated image classification model is used to classify images into multiple classes based on errors that occur. In this case, the calibrated image classification model is a multi-class classification model for classifying error types.

[0032] Based on the classification, it is possible to infer, for example, in which manufacturing process and / or in which part of the manufacturing equipment the classified error type occurs. To ensure reliable classification, it is necessary to ensure that the calibrated image classification model functions accordingly. For example, if the calibrated image classification model confuses specific classes or can no longer distinguish or identify specific classes, such inference results may no longer be valid. If the calibrated image classification model is unable to reliably distinguish between classes, this may further suggest that the images the model observes during use are no longer similar to the images on which the model was trained. In both cases, the model's functioning and performance are called into question.

[0033] Use of the above method to monitor a calibrated image classification model for an automated optical inspection AOI, for example, a calibrated image classification model used for automated optical inspection of components in component manufacturing, can solve the above-mentioned problems in component manufacturing and inspection.

[0034] A further embodiment relates to a system, e.g., a manufacturing facility, including a computing device configured to perform an automated optical inspection AOI using a calibrated image classification model, e.g., using a calibrated image classification model used for automated optical inspection of components in their manufacture, the calibrated image classification model being a multi-class model, the calibrated image classification model being trained to classify errors according to error types, the error types corresponding to respective classes in the calibrated image classification model, and the system including an apparatus according to the above embodiment.

[0035] Further advantages will become apparent from the description and the accompanying drawings. Exemplary embodiments of the invention are shown in the drawings and are explained in more detail in the following description, where the same reference signs in different drawings respectively designate identical elements or at least functionally equivalent elements. In describing an individual drawing, reference may also be made, where appropriate, to elements in other drawings. [Brief explanation of the drawings]

[0036] [Figure 1] FIG. 1 illustrates steps of a method for monitoring and evaluating an image classification model. [Figure 2] 2 is a schematic diagram of a visual output of a human-machine interface of an apparatus configured to implement the method of FIG. 1; DETAILED DESCRIPTION OF THE INVENTION

[0037] FIG. 1 illustrates the steps of a method 100 for monitoring and evaluating a calibrated image classification model.

[0038] The calibrated image classification model is configured to classify images into the corresponding class k of the image classification model, for example, after being trained with an appropriate training data set. The classification is performed, for example, based on a predicted softmax distribution, where each image is classified into the class for which the softmax distribution has the largest softmax value.

[0039] The method 100 includes the following steps.

[0040] That is, the method 100 includes a step 110 of classifying images using an image classification model, which includes predicting a softmax distribution. The classification into each class k is performed based on the predicted softmax value p, where k=argmax k p(y=k|x'), i.e., the image is classified into the class k that contains the highest softmax value in the predicted softmax distribution of that image.

[0041] The method 100 includes determining 120, for each image classified using the calibrated image classification model, a minimum class-dependent agreement value based on a predicted softmax distribution for the respective image and a characteristic softmax distribution of the class of the image classification model, the minimum class-dependent agreement value representing the class that deviates least from the characteristic softmax distribution of the class of the image classification model according to the predicted softmax distribution.

[0042] The method 100 includes generating 130 a visual output on a human-machine interface that enables a user to analyze the performance of the image classification model, the visual output including a per-class representation for each class of the image classification model. By per-class representation, it is meant that a separate diagram is generated and displayed for each class of the image classification model.

[0043] 2 shows an example diagram of a class representation for one class of an image classification model, which in this example is referred to as "Class 4."

[0044] In the class-by-class display for class 4, each image classified into class 4 using the image classification model is displayed as one data point (see the points shown in FIG. 2). That is, the class-by-class display for class 4 includes a plurality of data points, i.e., a data point equal to the number of images classified into class 4 using the image classification model. Each data point is displayed as a data point in a diagram in the class-by-class display of the visual output, and each data point is plotted as the softmax value of the images of each class over the corresponding class-dependent agreement value KL in the class 4 diagram shown in FIG. 2. That is, the diagram shows each data point as the softmax value of the images underlying that data point for each class over the corresponding class-dependent agreement value.

[0045] The visual output provides information about how high the corresponding class-dependent agreement value is for each softmax value of the underlying image for each data point in the class-specific representation for each class. The smaller the class-dependent agreement value, the more reliably the classification of each displayed class works for the underlying image for the data point. For example, classification for data points located further to the left in the diagram in Figure 2 works more reliably than classification for data points located further to the right.

[0046] Within the plot, each data point is displayed in color. The color of the color display is based on the class for which the image underlying the data point has the smallest class-dependent agreement value. The image does not necessarily have to have the smallest class-dependent agreement value for the class classified based on the highest softmax value. Note that the color display is simplified in Figure 2 due to the black and white representation.

[0047] In the diagram of Figure 2, data points with the lowest class-dependent agreement value for class 4 are displayed in a color that indicates class 4. Data points with the lowest class-dependent agreement value for class 5 are displayed in a color that indicates class 5. Data points with the lowest class-dependent agreement value for class 6 are displayed in a color that indicates class 6. For better understanding, the data points are shown in Figure 2 outlined by lines labeled 4, 5, and 6 according to their respective colors, so that color correspondences can be distinguished in the black and white drawings herein. These lines are not envisioned in the visual output of the present invention, nor are they required due to the color nature of the display.

[0048] 2, data points in a color indicating one of classes 1 to 3 are not displayed in this example, which means that in this case, none of the images classified into class 4 have a minimum class-dependent match value with one of classes 1 to 3.

[0049] In the diagram according to FIG. 2, two bends occur in the layout. These bends are labeled 20 and 22. At these bends, it is possible to identify a color change in the color display of the data points. The bends with color changes occur in the layout of the diagram of the class display for the data points, where the highest softmax score for the image underlying the data points corresponds to class 4 of the respective class display, but the softmax distribution is closer to the other classes, i.e., the class for the image underlying the data points has the smallest class-dependent agreement value. At bend 20, it is possible to identify a change between the color of class 4 and the color of class 6. At bend 22, it is possible to identify a change between the color of class 4 and the color of class 5. For example, a bend with color changes in the diagram can indicate that in some images, the model is unable or has difficulty in unambiguously distinguishing these classes according to the color change when classifying them.

[0050] According to the present method, it is assumed that, based on the selection of a data point in each class-specific representation of the visual output, the image underlying the data point is displayed in the visual output on the human-machine interface (step 140). That is, by selecting a data point, the image underlying the data point is displayed. This selection may be performed, for example, by a corresponding user input via a suitable input unit of the human-machine interface. Suitable input units are, for example, input devices such as a computer mouse, keyboard, trackpad, and / or touchscreen. Selecting a data point may include, for example, moving or holding a mouse pointer or finger over the data point in the sense of a mouse-over effect. In this way, the image underlying each data point becomes intuitively and quickly accessible to the user. This allows the user to quickly and efficiently gain insight into which specific image is underlying each data point.

[0051] The display of data points in the diagram in the class-by-class representation of the visual output, and the resulting arrangement and color display, allow for a clear and intuitive display of the data points, allowing a user to intuitively select which specific image they want to display by selecting the corresponding data point. The resulting arrangement and color display of data points in the diagram are suitable for suggesting which images in the model cause or may cause difficulties and should therefore be displayed to gain insight. That is, the display of data points conveys information to the user that enables the user to efficiently perform interactive exploration and selection of images.

[0052] Thus, the user can gain insight into which specific images the above-mentioned difficulties arise in classification. For example, the user can display desired images by sequentially selecting data points, thereby quickly and easily accessibly discovering similarities and / or differences between the displayed images. Thus, the user can personally investigate the images the model is observing during use, and thus gain a better understanding of the difficulties and / or problems of the image classification model, particularly those that could not be predicted at the time of training.

[0053] According to one embodiment, determining the minimum class-dependent match value for each image is based on the Kullback-Leibler divergence

number

number

[0054] According to one embodiment, it is assumed that the characteristic softmax distribution of the image classification model is the average softmax distribution of the classes of the validation data set and / or the training data set of the image classification model. In this case, when the method comprises the step of determining the characteristic softmax distribution, i.e. the average softmax distribution of the classes of the validation data set of the image classification model

number

Claims

1. 1. A computer-implemented method (100) for monitoring and evaluating a calibrated image classification model, comprising: The method comprises: classifying an image using the calibrated image classification model (110), which includes predicting a softmax distribution; determining (120) for each image classified using the calibrated image classification model a minimum class-dependent match value based on a predicted softmax distribution for the respective image and a characteristic softmax distribution of the class of the calibrated image classification model, wherein determining the minimum class-dependent match value for each image is performed by comparing the predicted softmax distribution for the respective image with the characteristic softmax distribution of the class of the calibrated image classification model based on Kullback-Leibler divergence; generating (130) a visual output on a human-machine interface that allows a user to analyze the performance of the calibrated image classification model, the visual output including a class-by-class representation for each class of the calibrated image classification model; Including, In each class representation, each image classified into a class using the calibrated image classification model is displayed as a single data point; Each data point is plotted as a softmax value of the images of each class over a corresponding class-dependent agreement value; Each data point is displayed in color, A method wherein a color is selected based on the class of the minimum class-dependent match value of the respective image.

2. Based on the selection of a data point in each of the class-based representations of the visual output, an image underlying the data point is displayed in the visual output on the human-machine interface (140). The method of claim 1.

3. deriving at least one evaluation of the calibrated image classification model based on the visual output; The evaluation is identifying at least one class for which the classification provided by the calibrated image classification model should be checked; and / or Identifying at least one data point, in particular an image underlying said data point, for which a classification provided by said calibrated image classification model should be checked. Including, 3. The method according to claim 1 or 2.

4. Based on the evaluation of the calibrated image classification model, a) training the calibrated image classification model with a new training data set; and / or b) revising the class definitions of the classes for which the classifications provided by the calibrated image classification model are to be checked; At least one of the following is implemented: The method of claim 3.

5. the characteristic softmax distribution of the calibrated image classification model is the average softmax distribution of the classes of a validation dataset and / or a training dataset of the calibrated image classification model; 5. The method according to any one of claims 1 to 4.

6. the characteristic softmax distribution of the calibrated image classification model is determined based on softmax distributions recorded during development of the model. The method of claim 1.

7. The method for monitoring and evaluating a calibrated image classification model is implemented to classify images in real time during the use of the calibrated image classification model.

7. The method according to any one of claims 1 to 6.

8. 1. An apparatus for monitoring and evaluating a calibrated image classification model, comprising: the device includes a computing device; the computing device is configured to execute machine-readable instructions that, when executed by the computing device, are capable of performing the method of any one of claims 1 to 7; the device includes a human-machine interface; By implementing the method according to any one of claims 1 to 7, it is possible to cause a visual output on the human-machine interface, the visual output includes a class-by-class representation for each class of the calibrated image classification model; In each class representation, each image classified into a class using the calibrated image classification model can be displayed as a single data point; Each data point may be plotted as the highest softmax value of the images of said respective class over the corresponding class-dependent agreement value; Each data point can be displayed in color, The apparatus is operable to select a color based on the class of the minimum class-dependent match value of the respective image.

9. and based on the selection of a data point in each of the class-based representations of the visual output, an image underlying the data point can be displayed in the visual output on the human-machine interface.

9. The apparatus of claim 8.

10. 1. Use of a method for monitoring a calibrated image classification model for automated optical inspection (AOI), for example a calibrated image classification model used for automated optical inspection of a component in its manufacture, comprising: the calibrated image classification model is a multi-class model; the calibrated image classification model is trained to classify errors according to error type; The use of the method, wherein an error type is associated with each class of the calibrated image classification model.

11. A system, e.g., a manufacturing facility, comprising: The system includes a computing device; the computing device is configured to perform automated optical inspection (AOI) using a calibrated image classification model, for example, using a calibrated image classification model used for automated optical inspection of a component in its manufacture; the calibrated image classification model is a multi-class model; the calibrated image classification model is trained to classify errors according to error type; an error type is associated with each class of the calibrated image classification model; The system comprises an apparatus according to claim 8 or 9.

Images