Method and apparatus for monitoring and evaluating image classification model

Abstract

The invention relates to a method and a device for monitoring and evaluating a calibrated image classification model, in particular for automated optical inspection, for example in the field of component production, for identifying defects.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to a computer-implemented method and apparatus for monitoring and evaluating an image classification model. Further embodiments relate to the use of the method and to a production facility comprising the apparatus. BACKGROUND

[0002] In the field of production, for example, it is sought in the field of automatic optical inspection (AOI) to monitor the proper functioning of the used image classification models and to evaluate the performance of these models. For the monitoring and performance evaluation, a dataset with class labels is usually required. However, at the point in time when the image classification model is used, labeled data is usually not available. Therefore, the monitoring is a challenge.

[0003] For example, problems arise when the model is used to classify images that are different or have a very low or even no similarity to those on which the model was trained. The classification of images that are different to those on which the model was trained is called out-of-distribution detection (OOD). Known methods for identifying OOD images require access to intermediate layers of the model and are therefore complex in implementation and cannot be intuitively employed in evaluation. SUMMARY

[0004] To solve this problem, a computer-implemented method according to claim 1 and an apparatus according to claim 9 are proposed.

[0005] One embodiment relates to a computer-implemented method for monitoring and evaluating a calibrated image classification model, wherein the method comprises:

[0006] classifying an image using the calibrated image classification model, which comprises predicting a Softmax distribution;

[0007] and, for the respective image classified using the calibrated image classification model, determining a class-related minimum match value based on the class-specific Softmax distribution of the calibrated image classification model and the predicted Softmax distribution for the respective image and

[0008] A visualization output is generated on a human-machine interface, which enables the user to analyze the performance of a calibrated image classification model. The visualization output includes a klassenweise representation of the corresponding category for the calibrated image classification model, wherein each image classified into the corresponding category using the calibrated image classification model is represented as a data point in the klassenweise representation, wherein the corresponding data point is plotted as the Softmax value of the image of the corresponding category on the associated category-related match value, and wherein the corresponding data point is represented by color, wherein the color is selected based on the category of the minimum category-related match value of the corresponding image.

[0009] According to this method, monitoring and evaluation are performed based on the predicted Softmax distribution, that is, based on the Softmax output, i.e., the class probabilities predicted by the classifier of the calibrated image classification model.

[0010] For each image, a minimum matching value associated with the category is determined based on the feature softmax distribution of the calibrated image classification model and the predicted softmax distribution. The minimum matching value associated with the category specifies the category for which the deviation between the predicted softmax distribution and the feature softmax distribution of the calibrated image classification model is minimized.

[0011] Further, the visualization output generated on the human-machine interface enables the user to analyze the performance of the calibrated image classification model. To this end, the visualization output includes a class-specific representation for each category of the calibrated image classification model. In the corresponding class-specific representation, each image classified into the corresponding category using the calibrated image classification model is represented as a data point, thus the corresponding class-specific representation contains multiple data points, i.e., the number of images classified into the corresponding category using the calibrated image classification model. Based on the predicted Softmax value p, the image is classified into the corresponding category k using k = argmaxkp(y = k|x'). In the class-specific representation of the visualization output, the corresponding data point is represented as a data point in a chart, where the corresponding data point is plotted as the Softmax value of the image in the corresponding category on the associated category-related match value. Therefore, the chart thus displays the data point as the Softmax value of the image in the corresponding category on which the data point is based, on the associated category-related match value. Class-specific representation means that a separate chart is generated and displayed for each category of the calibrated image classification model. In the chart, the corresponding data points are represented by color. The color representation is based on the following category, where the data point is based on the image with the minimum matching value associated with that category.

[0012] For example, in the class representation of the first category, images classified into the first category based on the predicted Softmax distribution using a calibrated image classification model are represented as data points. This means that the first category has the highest Softmax value in the predicted Softmax distribution for these images. The corresponding data points are then plotted in a graph as the Softmax value of the first category, i.e., the match value for the first category, on the associated category-related match value.

[0013] The visualization output provides information in the class representation of the corresponding category regarding the size of the class-related match value associated with the corresponding softmax value of the image on which the corresponding data point is based. The smaller the class-related match value, the more reliable the classification of the corresponding displayed category.

[0014] Each data point is represented by a matching value associated with a category. If the image on which the data point is based has the minimum matching value associated with a first category, then the data point is shown in a first color indicating the first category. If the image on which the data point is based has the minimum matching value associated with a second category, then the data point is shown in a second color indicating the second category. Processing continues accordingly with further categories and colors.

[0015] Therefore, if the highest Softmax score of the image on which the data point is based corresponds to the corresponding class representation, but the Softmax distribution is closer to another class—that is, the class on which the image on which the data point is based has the smallest matching value with respect to that class—then the visualization output will provide relevant information in the class representation of the corresponding class. This information may indicate that the image on which the corresponding data point is based is difficult or impossible for the model to explicitly classify.

[0016] Furthermore, the representation in the above visualization output results in multiple data points from images that the model struggles to distinguish being arranged linearly in the chart with an inflection point, where the color changes. This information may indicate that the model has difficulty distinguishing the image category on which the corresponding data points are based based on the color representation of the data points.

[0017] The way data points are represented in the class-based visualization output of the chart, the resulting arrangement, and the color representation allow users to directly obtain information about the internal state of the technical system—that is, information about the functionality and performance of the calibrated image classification model—based on the presented information. Based on this representation and the resulting information about the internal state, users can take appropriate actions and / or interact with the calibrated image classification model, for example, to improve the model's functionality and performance and avoid functional failures.

[0018] In cases with a large number of categories, it can be stipulated that, particularly for a better overview, not every category is shown in terms of the minimum matching value associated with that category in the class representation. For example, several data points can be identified for each minimum matching value associated with a category, and then only those categories whose frequency exceeds a defined threshold, for example, 1% of the number of identified data points, are shown. Alternatively, multiple separate visualizations of the class representation can be created, where subsets of these categories are shown in terms of the minimum matching value associated with that category.

[0019] According to one implementation, based on the selection of a data point in a corresponding categorized representation of the visualization output, the image on which that data point is based is displayed in the visualization output on the human-machine interface. Therefore, the selection of a data point causes the display of the image on which that data point is based. For example, this selection can be made via appropriate input devices of the human-machine interface through corresponding user input. Appropriate input devices are, for example, input devices such as a computer mouse, keyboard, touchpad, and / or touchscreen. The selection of a data point can, for example, include moving or holding a mouse pointer or finger over or on a data point, i.e., a mouse hover effect. In this way, the user can access the image on which the corresponding data point is based in an intuitive and fast manner. Therefore, the user can quickly and efficiently gain a deeper understanding of the specific image on which the corresponding data point is based.

[0020] The way data points are represented in the categorized output of the chart, along with their arrangement and color scheme, makes the data point representation easy to understand and intuitive, allowing users to make intuitive choices about which specific images they want to display by selecting the appropriate data points. The arrangement and color scheme of the data points in the chart are suitable for indicating which images contribute to or may contribute to difficulties in the model and therefore should be displayed for deeper understanding. Thus, the representation of the data points conveys information to the user, enabling them to efficiently perform interactive image searches and selections.

[0021] Therefore, users gain in-depth understanding of which specific images exhibit the described classification difficulties. For example, by selecting data points, users can display the desired images one by one and quickly and easily identify the similarities and / or differences between the displayed images. Thus, users can examine the images the model encounters during use themselves and thereby better understand the difficulties and / or problems of the image classification model, especially those difficulties and / or problems that were unforeseen during training.

[0022] According to one implementation, at least one evaluation of a calibrated image classification model is derived based on visualization output, wherein the evaluation includes: identifying at least one category for which the calibrated image classification model provides a classification to be tested; and / or identifying at least one data point, and in particular the image on which the data point is based, for which the calibrated image classification model provides a classification to be tested. The classification to be tested is, for example, understood as an unsatisfactory classification or a questionable classification. An unsatisfactory classification or a questionable classification is, for example, understood as: difficulty in classification associated with that category. For example, two or more categories may also be identified for which the model cannot clearly distinguish or has difficulty distinguishing during the classification process.

[0023] For example, a high category-related match value may indicate that there is classification difficulty associated with that category. A low category-related match value may indicate that there is no classification difficulty associated with that category.

[0024] For example, a color-changing inflection point in a chart might indicate that the model cannot explicitly or easily distinguish these categories based on color changes during the classification process. Such color-changing inflections occur in charts representing data points by category, where, in the case of these data points, the highest Softmax score of the image on which the data point is based corresponds to the corresponding category, but the Softmax distribution is closer to another category—the category on which the image on which the data point is based has the smallest matching value related to that category.

[0025] According to one implementation method, based on the evaluation of the calibrated image classification model, at least one of the following steps is performed:

[0026] a) Train or retrain the calibrated image classification model based on the modified training dataset, and / or b) Modify the category definitions for at least the following categories to which the calibrated image classification model provides a classification to be tested: The modified training dataset can be improved relative to the previously used training dataset, for example, by adding or deleting images. In particular, the modified training dataset may contain one or more images of categories to which the model has difficulty. The category definitions are determined, for example, by domain experts (e.g., experts in the field of AOI) based on the defects to be identified in the images. Such category definitions can often be somewhat arbitrary. Therefore, changing the category definitions can lead to improved model performance.

[0027] According to one implementation, the minimum class-related match value for a given image is determined based on the Kullback-Leibler divergence by comparing the feature softmax distribution of the image classification model's class with the predicted softmax distribution of the corresponding image. Therefore, using the Kullback-Leibler divergence can be advantageous because: only model prediction is needed for the determination, without requiring information from intermediate layers of the model that is typically unavailable during model usage. The Kullback-Leibler divergence is described, for example, in arXiv:1911.11132, as a KL matching score used to identify images whose softmax distribution differs from the softmax distribution of images that should typically be observed during training. In this regard, the KL matching score is used to determine the OOD (Out of Detail) attribute of an image.

[0028] Instead of the Kullback-Leibler divergence, other OOD values ​​available at the image level can be used. For example, the minimum class-related match value for a given image can be determined based on the cosine similarity of the mean of intra-class features (arXiv:2306.14920) or the distance based on the activation histogram (arXiv:2309.04837).

[0029] According to one implementation, the feature softmax distribution of the image classification model is the average softmax distribution of the classes in the validation dataset and / or training dataset of the calibrated image classification model. In this case, it may prove advantageous to make the method include the step of determining the feature softmax distribution, i.e., in the form of the average softmax distribution of the classes in the validation dataset of the calibrated image classification model.

[0030] According to another implementation, the feature softmax distribution of the calibrated image classification model can be determined based on the softmax distribution recorded during model development. Therefore, the image distribution is detected during development. This can be advantageous, for example, when no validation set and / or training dataset is available.

[0031] According to one implementation, the method for monitoring and evaluating a calibrated (kalibriert) image classification model is performed in real time during the classification of images using the calibrated image classification model.

[0032] A further embodiment relates to an apparatus for monitoring and evaluating a calibrated image classification model, wherein the apparatus includes a computing device designed to execute machine-readable instructions that, when executed, can perform the method according to the embodiment, and wherein the apparatus includes a human-machine interface on which visual output can be caused by executing the method according to the embodiment, wherein the visual output includes a class representation of a corresponding category for the calibrated image classification model, and wherein in the corresponding class representation, each image classified into the corresponding category using the calibrated image classification model can be represented as a data point, wherein the corresponding data point can be plotted as the maximum Softmax value of the image of the corresponding category on an associated category-related matching value, and wherein the corresponding data point can be represented by color (farbig), wherein the color can be selected based on the category of the minimum category-related matching value of the corresponding image.

[0033] According to one implementation, based on the selection of data points in the corresponding class representation of the visualization output, the image on which the data point is based can be displayed in the visualization output on the human-machine interface.

[0034] Further embodiments relate to the use of the method for monitoring a calibrated image classification model for automated optical inspection (AOI), such as a calibrated image classification model for automated optical inspection of components in component manufacturing, wherein the calibrated image classification model is a multi-class model, wherein the calibrated image classification model is trained to classify defects according to defect types, wherein defect types are assigned to corresponding categories in the calibrated image classification model.

[0035] In the manufacturing industry, it is known that optical inspection is performed on finished components to ensure the quality of the production process. For example, undesirable damage such as scratches, contamination, breakage, deformation, and discoloration can be identified. Optical inspection can be performed automatically, i.e., automated optical inspection (AOI), particularly to improve efficiency and reduce costs. For example, a calibrated image classification model is used, which classifies images into multiple categories based on the defects produced. In this case, the calibrated image classification model is a multi-class classification model used to classify defect types.

[0036] For example, based on classification results, it can be inferred which parts and / or which production steps of the production facility the classified defect type appears in. For reliable classification, it is essential to ensure that the calibrated image classification model functions properly. For instance, if the calibrated image classification model confuses a particular category, or can no longer distinguish or identify a particular category, such conclusions may no longer be valid. If the calibrated image classification model cannot reliably distinguish categories, this may also indicate that the images the model sees during use are no longer similar to the images on which it was trained. In both cases, the model's methodology and performance are questionable.

[0037] Applying the above method to monitor calibrated image classification models used for automated optical inspection (AOI), such as those used for automated optical inspection of components in component manufacturing, can eliminate the aforementioned problems in component manufacturing and inspection.

[0038] A further embodiment relates to a system, such as a production facility, comprising: a computing device designed to perform automated optical inspection (AOI) using a calibrated image classification model, such as a calibrated image classification model used for automated optical inspection of components in component production, wherein the calibrated image classification model is a multi-class model, wherein the calibrated image classification model is trained to classify defects according to defect types, wherein defect types are assigned to corresponding categories in the calibrated image classification model, and wherein the system includes means according to the embodiment. Attached Figure Description

[0039] Further advantages arise from the specification and accompanying drawings. Embodiments of the invention are shown in the drawings and explained in more detail in the following description. Here, the same reference numerals in different figures denote the same or at least functionally equivalent elements. In describing the various figures, reference is also made to elements in the other figures. In illustrative form:

[0040] Figure 1 The steps for monitoring and evaluating image classification models are shown, as well as...

[0041] Figure 2 It shows that it is designed to perform according to Figure 1 A schematic diagram of the visual output of the human-machine interface of the device. Detailed Implementation

[0042] Figure 1 The steps of a method 100 for monitoring and evaluating a calibrated image classification model are shown in the figure.

[0043] The calibrated image classification model is designed to, for example, be trained on a suitable training dataset to classify images into corresponding categories k of the image classification model. For example, the classification is based on a predicted Softmax distribution, where the corresponding image is classified into a category for which the Softmax distribution has the largest Softmax value.

[0044] The method 100 includes the following steps:

[0045] Step 110, which classifies an image using an image classification model, includes: predicting the Softmax distribution. Based on the predicted Softmax value p, k = argmax... k p(y=k|x′) is classified into the corresponding category k. Therefore, the image is classified into category k, which has the highest softmax value in the predicted softmax distribution of the image.

[0046] The method 100 includes step 120, which involves determining a minimum matching value associated with a category for a given image classified using a calibrated image classification model, based on the feature softmax distribution of the category in the image classification model and the predicted softmax distribution for the given image. The minimum matching value associated with a category specifies the category for which the deviation from the feature softmax distribution of the category in the image classification model is minimized, based on the predicted softmax distribution.

[0047] The method 100 includes step 130, which generates a visualization output on a human-computer interface, the visualization output enabling a user to analyze the performance of an image classification model, wherein the visualization output includes a class-specific representation of the image classification model for each category. Class-specific representation means that a separate chart is generated and displayed for each category of the image classification model.

[0048] Figure 2 An example diagram showing the class representation of categories for an image classification model is provided. The category is represented as "Category 4" in the example.

[0049] In the class-based representation of category 4, each image classified into category 4 using the image classification model is represented as a data point, see [link to relevant documentation]. Figure 2 These points are shown in the diagram. The class representation of category 4 contains multiple data points, i.e., the number of images classified into category 4 using the image classification model. The corresponding data points are represented as data points in the chart in the visualization output, where... Figure 2In the chart for category 4 shown, the corresponding data points are plotted as the softmax value of the corresponding category's image on the associated category-related match value KL. Therefore, the chart displays each data point as the softmax value of the corresponding category's image on the associated category-related match value.

[0050] The visualization output provides information in the class-specific representation of the corresponding category regarding the size of the class-related match value associated with the corresponding softmax value of the image on which the corresponding data point is based. The smaller the class-related match value, the more reliable the classification of the corresponding indicated category for the image on which that data point is based. For example, for... Figure 2 In the chart, the classification of data points on the left is more reliable than that of data points on the right.

[0051] In the chart, the corresponding data points are represented by color. The color is based on the category to which the data point is based, and the image it is based on has the minimum matching value associated with that category. The image does not necessarily have to have the minimum matching value associated with the category classified based on the highest Softmax value. It should be noted that... Figure 2 The color representations in the image are simplified due to the use of a black and white image.

[0052] exist Figure 2 In the chart, data points indicating category 4 are shown in color, and these data points have the minimum matching value associated with category 4. Data points with the minimum matching value associated with category 5 are shown in color indicating category 5. Data points with the minimum matching value associated with category 6 are shown in color indicating category 6. For better understanding, Figure 2 The data points are surrounded by lines labeled 4, 5, and 6 according to their corresponding colors, so as to indicate the color assignment in the current black and white representation. These lines are not provided in the visualization output according to the invention, and since color representation is not necessary.

[0053] According to Figure 2 In the chart shown in this example, no data points indicating the color for one of categories 1 through 3 are displayed. This means that in this case, no image classified as category 4 has the minimum matching value associated with one of categories 1 through 3.

[0054] According to Figure 2In the chart, two inflection points are derived from this arrangement. They are labeled 20 and 22. At these inflection points, a color change in the color representation of the data points can be identified. The inflection points with color changes appear in the arrangement of the chart for data points represented by categories, where, in the case of these data points, the highest Softmax score of the image on which the data point is based corresponds to category 4 in the corresponding category representation, but the Softmax distribution is closer to another category, namely the category on which the image on which the data point is based has the minimum matching value related to the category. At inflection point 20, a change between the color for category 4 and the color for category 6 can be identified. At inflection point 22, a change between the color for category 4 and the color for category 5 can be identified. For example, the inflection points with color changes in the chart may indicate that, in the case of some images, the model cannot clearly distinguish or has difficulty distinguishing these categories based on the color changes in the classification.

[0055] According to this method, step 140 involves displaying the image on which the selected data point is based in the visualization output on the human-machine interface, based on the selection of the data point in the corresponding categorized representation of the visualization output. Therefore, the selection of a data point results in the display of the image on which the data point is based. For example, this selection can be made via appropriate input devices of the human-machine interface through corresponding user input. Appropriate input devices include, for example, computer mice, keyboards, touchpads, and / or touchscreens. The selection of a data point can, for example, include moving or holding a mouse pointer or finger over or on the data point, i.e., a mouse hover effect. In this way, the user can access the image on which the corresponding data point is based in an intuitive and fast manner. Therefore, the user can quickly and efficiently gain a deeper understanding of the specific image on which the corresponding data point is based.

[0056] The way data points are represented in the categorized output of the chart, along with their arrangement and color scheme, makes the data point representation easy to understand and intuitive, allowing users to make intuitive choices about which specific images they want to display by selecting the appropriate data points. The arrangement and color scheme of the data points in the chart are suitable for indicating which images contribute to or may contribute to difficulties in the model and therefore should be displayed for deeper understanding. Thus, the representation of the data points conveys information to the user, enabling them to efficiently perform interactive image searches and selections.

[0057] Therefore, users gain in-depth understanding of which specific images exhibit the described classification difficulties. For example, by selecting data points, users can display the desired images one by one and quickly and easily identify the similarities and / or differences between the displayed images. Thus, users can examine the images the model encounters during use themselves and thereby better understand the difficulties and / or problems of the image classification model, especially those difficulties and / or problems that were unforeseen during training.

[0058] According to one implementation method, the softmax distribution of features of the categories of the image classification model is compared. Based on the predicted Softmax distribution of the corresponding image, determine the minimum class-related matching value min for the corresponding image using the Kullback-Leibler divergence. k KL[p(y|x)||d k Therefore, using Kullback-Leibler divergence can be advantageous because: only model predictions are needed for determination, without requiring information from intermediate layers of the model that is typically unavailable during model usage. Kullback-Leibler divergence is described, for example, in arXiv:1911.11132, as a KL matching score used to identify images whose softmax distribution differs from the softmax distribution of images that should typically be observed during training. In this regard, the KL matching score is used to determine the OOD attribute of an image.

[0059] According to one implementation, the feature softmax distribution of an image classification model is the average softmax distribution of the classes in the validation dataset and / or training dataset of the image classification model. In this case, it may prove advantageous to include a step for determining the feature softmax distribution, i.e., the average softmax distribution of the classes in the validation dataset of the image classification model. In the form of.

Claims

1. A computer-implemented method (100) for monitoring and evaluating a calibrated image classification model, wherein the method comprises: The image is classified using the calibrated image classification model (110), which includes predicting the Softmax distribution; For a given image classified using a calibrated image classification model, the minimum matching value associated with the class is determined (120) based on the feature softmax distribution of the class in the calibrated image classification model and the softmax distribution predicted for the given image. The minimum matching value of the corresponding image related to the category is determined based on the Kullback-Leibler divergence by comparing the feature softmax distribution of the category of the calibrated image classification model with the predicted softmax distribution of the corresponding image. And generate (130) visualization output on the human-machine interface, the visualization output enabling the user to analyze the performance of the calibrated image classification model, wherein the visualization output includes a class representation of the corresponding category for the calibrated image classification model, and wherein in the corresponding class representation, each image classified into the corresponding category using the calibrated image classification model is represented as a data point, wherein the corresponding data point is plotted as the Softmax value of the image of the corresponding category on the associated category-related matching value, and wherein the corresponding data point is represented by color, wherein the color is selected based on the category of the minimum category-related matching value of the corresponding image.

2. The method of claim 1, wherein the image on which the data points are based is displayed (140) in the visualization output on the human-machine interface based on the selection of data points in the corresponding class representation of the visualization output.

3. The method according to any one of claims 1 or 2, wherein at least one evaluation of the calibrated image classification model is derived based on the visualization output, wherein the evaluation comprises: Identify at least one of the following categories, for which a calibrated image classification model provides a classification to be tested; And / or identify at least one data point, and in particular the image on which the data point is based, for which the calibrated image classification model provides a classification to be tested for the data point and, in particular, the image on which the data point is based.

4. The method of claim 3, wherein, based on the evaluation of the calibrated image classification model, at least one of the following steps is performed: a) training the calibrated image classification model on a new training dataset, and / or b) modifying the category definitions for at least the following categories for which the calibrated image classification model provides a classification to be tested.

5. The method according to any one of the preceding claims, wherein the feature softmax distribution of the calibrated image classification model is the average softmax distribution of the categories of the validation dataset and / or training dataset of the calibrated image classification model.

6. The method of claim 1, wherein the feature softmax distribution of the calibrated image classification model is determined based on the softmax distribution recorded during model development.

7. The method according to any one of the preceding claims, wherein, The method for monitoring and evaluating the calibrated image classification model is performed in real time during the classification of images using the calibrated image classification model.

8. An apparatus for monitoring and evaluating a calibrated image classification model, wherein the apparatus includes a computing device designed to execute machine-readable instructions that, when executed, enable the execution of the instructions to perform the method according to any one of claims 1 to 7, and wherein the apparatus includes a human-machine interface, wherein performing the method according to any one of claims 1 to 7 can cause visual output on the human-machine interface, wherein the visual output includes a class representation of a corresponding category for the calibrated image classification model, and wherein in the corresponding class representation, each image classified into the corresponding category using the calibrated image classification model can be represented as a data point, wherein the corresponding data point can be plotted as the maximum Softmax value of the image of the corresponding category on an associated category-related matching value, and wherein the corresponding data point can be represented by color, wherein the color can be selected based on the category of the minimum category-related matching value of the corresponding image.

9. The apparatus according to claim 8, wherein, Based on the selection of data points in the corresponding class representation of the visualization output, the image on which the data points are based can be displayed in the visualization output on the human-machine interface.

10. Use of a method for monitoring a calibrated image classification model for automated optical inspection (AOI), wherein the calibrated image classification model is, for example, a calibrated image classification model for automated optical inspection of said components in component production, wherein the calibrated image classification model is a multi-class model, wherein the calibrated image classification model is trained to classify defects according to defect types, wherein defect types are assigned to corresponding categories of said calibrated image classification model.

11. A system, such as a production facility, comprising a computing device designed to perform automated optical inspection (AOI) using a calibrated image classification model, for example, a calibrated image classification model used for automated optical inspection of said components in component production, wherein said calibrated image classification model is a multi-class model, wherein said calibrated image classification model is trained to classify defects according to defect types, wherein defect types are assigned to corresponding categories of said calibrated image classification model, and wherein said system comprises means according to any one of claims 8 or 9.

Images