Image inspection device, classification correction method, and classification correction program

Abstract

To provide an image inspection device, a classification correction method, and a classification correction program that support appropriate corrections of classification labels associated with images used to train a classification model.SOLUTION: In an inspection system 100, an image inspection device 13 for performing image inspection using a classification model that classifies images of a subject to be inspected comprises an anomaly estimation unit 32, a presentation unit 36, and a change unit 38. The anomaly estimation unit 32 uses an anomaly detection model to estimate anomalies in each of a plurality of images associated with classification labels as teacher candidate images for the classification model that classifies the images of the subject to be inspected. The presentation unit 36 presents at least some of the plurality of images based on estimation results by the anomaly estimation unit 32. The change unit 38 changes the classification labels of images that have undergone a classification label change operation, among the at least some of the images presented by the presentation unit 36.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present disclosure relates to an image inspection device, a classification correction method, and a classification correction program. [Background technology]

[0002] To maximize the performance of a classification model obtained through machine learning, it is necessary to train the classification model using data of sufficient quality and quantity, but it takes a huge amount of time and effort for a person to manually check that all the data used for training is appropriate.

[0003] Therefore, Patent Document 1 proposes a technology that uses a classification model to estimate a classification score for each of multiple teacher candidate images, and presents the classification results to an operator by sorting the multiple teacher candidate images in order of lowest estimated classification score. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2020-24534 Summary of the Invention [Problem to be solved by the invention]

[0005] The technology described in Patent Document 1 above has the problem that when the performance of the classification model is low, for example, immediately after initial training of the classification model, it is not possible to properly sort the candidate images used to train the classification model, making it difficult to properly correct the classification labels.

[0006] The present disclosure provides techniques that can assist in the appropriate modification of classification labels associated with images used to train classification models. [Means for solving the problem]

[0007] An image inspection device according to one aspect of the present disclosure is an image inspection device that performs image inspection using a classification model that classifies images of an object to be inspected, and includes an anomaly estimation unit, a presentation unit, and a change unit. The anomaly estimation unit uses the anomaly detection model to estimate an anomaly in each of a plurality of images associated with a classification label as a teacher candidate image for the classification model that classifies the images of the object to be inspected. The presentation unit presents at least some of the plurality of images based on the estimation result by the anomaly estimation unit. The change unit changes the classification label of an image that has undergone a classification label change operation among the at least some of the images presented by the presentation unit. [Effects of the Invention]

[0008] The present disclosure can assist in appropriately modifying classification labels associated with images used to train a classification model. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram illustrating an example of a classification correction process according to the embodiment. [Figure 2] FIG. 2 is a block diagram showing an example of the configuration of an inspection system according to the embodiment. [Figure 3] FIG. 3 is a diagram showing the relationship between the distribution of features of a group of typical images of a class and the distribution of features of a group of abnormal images of that class in a high-dimensional feature space. [Figure 4] FIG. 4 is a diagram showing an example of an inspection screen displayed on a display device by a presentation unit in a processing unit of an image inspection device according to an embodiment. [Figure 5] FIG. 5 is a diagram showing an example of the relationship between the weight of each score by the sorting processing unit in the processing unit of the image inspection device according to the embodiment and the value of the evaluation index. [Figure 6] FIG. 6 is a diagram showing an example of a management screen displayed on a display device by a presentation processing unit in a processing unit of an image inspection device according to an embodiment. [Figure 7] FIG. 7 is a diagram showing an example of an additional information screen displayed on a display device by a presentation unit in a processing unit of an image inspection device according to an embodiment. [Figure 8] FIG. 8 is a diagram showing another example of the additional information screen displayed on the display device by the presentation unit in the processing unit of the image inspection device according to the embodiment. [Figure 9] FIG. 9 is a flowchart showing an example of information processing by the processing unit of the image inspection device according to the embodiment. [Figure 10] FIG. 10 is a flowchart showing an example of an appearance inspection process performed by the processing unit of the image inspection device according to the embodiment. [Figure 11] FIG. 11 is a flowchart showing an example of classification correction processing by the processing unit of the image inspection device according to the embodiment. [Figure 12] FIG. 12 is a flowchart showing an example of an evaluation score determination process performed by the processing unit of the image inspection device according to the embodiment. [Figure 13] FIG. 13 is a diagram illustrating an example of the hardware configuration of a processing unit in an image inspection device according to an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments of the image inspection device, classification correction method, and classification correction program disclosed in the present application will be described in detail with reference to the accompanying drawings. Note that the disclosed technology is not limited to these embodiments.

[0011] <1. An example of classification correction processing> FIG. 1 is a diagram showing an example of a classification correction process according to an embodiment. The classification correction process according to the embodiment is executed by an image inspection device described below, but may also be executed by a device other than an image inspection device.

[0012] In the example shown in FIG. 1, images IMA1 to IMA500, IMB1 to IMB20, and IMC1 to IMC10 are shown as a plurality of images associated with classification labels as training candidate images for a classification model that classifies images of the object to be inspected.

[0013] Each of the images IMA1 to IMA500, IMB1 to IMB20, and IMC1 to IMC10 is an image of an object to be inspected. The object to be inspected is an object to be inspected, such as a workpiece or product produced on a factory production line, but may also be another object.

[0014] Images IMA1 to IMA500 are images associated with class A as a classification label, and when images IMA1 to IMA500 are shown without being individually distinguished, they may be referred to as image IMA. In the example shown in Figure 1, the number of images IMA that are candidate teacher images is set to 500 for convenience, but the number of images IMA is not limited to the example shown in Figure 1.

[0015] Images IMB1 to IMB20 are images associated with class A as a classification label, and when images IMB1 to IMB20 are shown without being individually distinguished, they may be referred to as image IMB. In the example shown in Figure 1, the number of images IMB that are teacher candidate images is set to 20 for convenience, but the number of images IMB is not limited to the example shown in Figure 1.

[0016] Images IMC1 to IMC10 are images associated with class A as a classification label, and when images IMC1 to IMC10 are shown without being individually distinguished, they may be referred to as image IMC. In the example shown in Figure 1, the number of images IMC that are teacher candidate images is set to 10 for convenience, but the number of images IMB is not limited to the example shown in Figure 1. In the following, when images IMA, IMB, and IMC are shown without being individually distinguished, they may be referred to as image IM.

[0017] Class A is, for example, a non-defective product, and classes B and C are, for example, different types of defective products. The number of classifications by the classification model is not limited to three, A, B, and C, but may be four or more, or may be two.

[0018] The classification model is, for example, a model generated by machine learning, such as a convolutional neural network, but is not limited to such an example and may be, for example, an autoencoder or other model.

[0019] The image inspection device executes an anomaly estimation process to estimate an anomaly in each of a plurality of images IM associated with a classification label as a training candidate image for a classification model (step S1). In the anomaly estimation process, an anomaly in each of the plurality of images IM is estimated for each class based on the difference between the feature amount of a group of correct images of the class classified by the classification model and the feature amount of each of the plurality of images IM.

[0020] The correct image group for a class used in the anomaly estimation process is a typical image group, which is a group of images that includes a plurality of typical images as a plurality of images classified into that class. A typical image of a class is also called a typical image of the class or a representative image of the class, and is an image that exhibits the typical characteristics of the class.

[0021] Each image included in the group of typical images for class A is an image that shows typical features of class A, and each image included in the group of typical images for class B is an image that shows typical features of class B. Each image included in the group of typical images for class C is an image that shows typical features of class C.

[0022] The typical image group for a class is, for example, a group of images selected manually, but may also be a group of images including a plurality of images whose classification scores for that class are equal to or greater than a threshold, which are output from a classification model. Note that the correct image group for a class used in the anomaly estimation process is not limited to a typical image group, and may include, for example, images other than the typical images.

[0023] In the anomaly estimation process, for example, an anomaly is estimated for each of the multiple images IM based on the difference between the distribution of feature values ​​of the correct image group of classes classified by the classification model and the feature values ​​of each of the multiple images IM. The image inspection device estimates the anomaly of the image IM using PatchCore technology.

[0024] First, the image inspection device extracts the feature amounts of each correct image included in the group of correct images. The feature amounts of the correct images are extracted using, for example, an anomaly detection model, which is a pre-trained deep learning model such as Residual Network (ResNet). The feature amounts of the correct images are extracted, for example, for each patch. The feature amounts of the correct images are represented by a high-dimensional feature vector.

[0025] The image inspection device also extracts feature quantities for each image IM using the same extraction method as that for the target image. The image inspection device then compares the distribution of feature quantities for each target image included in the target image group with the feature quantities of the image IM for each patch, and calculates an anomaly score for each patch. The calculation of the anomaly score is performed using, for example, a nearest neighbor method, but is not limited to this example.

[0026] The image inspection device then calculates the degree of abnormality of each pixel based on the degree of abnormality of each patch. For example, the image inspection device calculates the degree of abnormality of each pixel in the image IM by assigning the degree of abnormality of each patch to a corresponding position in the image IM.

[0027] The image inspection device estimates an anomaly score as an index value indicating an abnormality in the image IM based on the degree of anomaly of each pixel. The anomaly score includes a first anomaly score indicating the maximum value of the degree of anomaly per pixel, and a second anomaly score indicating the proportion of areas with an anomaly degree equal to or greater than a threshold to the entire image. Note that the first anomaly score may be, for example, the average or median of the anomaly degrees of areas with an anomaly degree equal to or greater than a threshold.

[0028] In this way, the image inspection device can estimate an abnormality score as the abnormal state of each of the multiple images IM based on the difference between the distribution of features of the group of correct images of the class classified by the classification model and the features of each of the multiple images IM.

[0029] The image inspection device can also estimate whether or not an abnormality exists in the image IM based on the estimated abnormality score. For example, the image inspection device estimates that an abnormality exists in the image IM when the first abnormality score is equal to or greater than a threshold, when the second abnormality score is equal to or greater than a threshold, or when the score obtained by weighting the first abnormality score and the second abnormality score is equal to or greater than a threshold. An abnormality in the image IM is an abnormality in the appearance of the object being inspected shown in the image IM.

[0030] When the processing of step S1 is completed, the image inspection device presents at least some of the images IM based on the estimation result of step S1 (step S2). The presentation of the images IM is performed, for example, by displaying the images IM on a display device.

[0031] For example, the image inspection device sorts the multiple images IM into classes based on the abnormality scores estimated in step S1, and presents at least some of the images IM for each class based on the sorting results.

[0032] For example, the image inspection device sorts a plurality of images IM for each class in descending or ascending order of the first abnormality score, the second abnormality score, or the score obtained by weighted addition of the first abnormality score and the second abnormality score.

[0033] The image inspection device can also sort the multiple images IM by class based on the evaluation score calculated based on the anomaly score and the classification score, and present at least some of the images IM by class based on the sorting results. For example, the image inspection device sorts the multiple images IM by class in descending or descending order of the evaluation score.

[0034] In this case, the image inspection device estimates a classification score for each of the multiple images IM using a classification model. For example, the image inspection device inputs the images IM into the classification model and causes the classification model to output a classification score for each class. The image inspection device sorts the multiple images IM by class based on an evaluation score calculated based on the classification score and the anomaly score.

[0035] As an image inspection device, an evaluation score for each of a plurality of images is calculated by weighted addition of a plurality of scores, including, for example, a first score obtained by normalizing the first abnormality score, a second score obtained by normalizing the second abnormality score, and a third score obtained by normalizing the classification score.

[0036] For example, the image inspection device presents at least some of the images IM based on the anomaly scores or evaluation scores estimated in step S1. For example, the image inspection device presents images IM with a specific score equal to or greater than a threshold. The specific score may be, for example, an image IM with a first anomaly score equal to or greater than a threshold among the multiple images IM, an image IM with a second anomaly score equal to or greater than a threshold among the multiple images IM, a weighted sum of the first anomaly score and the second anomaly score, or an evaluation score.

[0037] The image inspection device presents a predetermined number of images IM from the sorted plurality of images IM in descending order of score as a portion of images IM for each class, or presents images IM from the plurality of images IM with scores below a threshold as a portion of images IM for each class.

[0038] In the example shown in Figure 1, among the multiple images IM, images IM with scores below a threshold are presented for each class. For example, among the multiple images IMA in class A, images IMA451, IMA28, and IMA124 with scores below the threshold are presented. Also, among the multiple images IMB in class B, image IMB12 with a score below the threshold is presented. Also, among the multiple images IMC in class C, image IMC3 with a score below the threshold is presented.

[0039] In addition, when the image inspection device estimates whether or not an image IM contains an abnormality based on the estimated abnormality score, it can also present images IM that are not estimated to contain an abnormality from multiple images IM by class.

[0040] Next, the image inspection device changes the classification labels of images that have undergone a classification label change operation among at least some of the images IM for each class presented in step S2 (step S3). This allows the image inspection device to support appropriate correction of classification labels associated with images used in training the classification model.

[0041] Next, the image inspection device performs training of the classification model using a plurality of images IM including the image IM whose classification label has been changed in step S3 (step S4). This enables the image inspection device to perform training of the classification model with high accuracy.

[0042] An example of the configuration of an inspection system including an image inspection device that performs classification correction processing according to the embodiment will be specifically described below.

[0043] <2. Inspection system configuration> 2 is a block diagram showing an example of the configuration of an inspection system 100 according to an embodiment. The inspection system 100 shown in FIG. 2 includes an imaging device 10, a display device 11, an operation device 12, and an image inspection device 13.

[0044] 2.1. Imaging device 10 The imaging device 10 is an image sensor that captures images of the objects to be inspected, and includes, for example, a CMOS (Complementary Metal Oxide Semiconductor). The imaging device 10 captures an image of each object to be inspected, and outputs information indicating the captured image of the object to the image inspection device 13.

[0045] The imaging device 10 is placed, for example, on a production line in a factory. The imaging device 10 is a wired camera that communicates with the image inspection device 13 via a wired connection, but may also be a wireless camera that can communicate with the image inspection device 13 wirelessly.

[0046] <2.2.Display device 11> The display device 11 is, for example, an LCD (Liquid Crystal Display) or an organic EL (Electro Luminescence) display.

[0047] <2.3. Operating device 12> The operation device 12 includes, for example, a keyboard, a mouse, a power button, etc. If the display device 11 is a touch panel display, the operation device 12 includes a touch panel. The operation device 12 is operated by, for example, an inspector O.

[0048] <2.4. Image inspection equipment 13> The image inspection device 13 is a device that performs image inspection using a classification model that classifies an image of an object to be inspected, and includes a storage unit 20 and a processing unit 21.

[0049] <2.4.1.Storage section 20> The storage unit 20 stores information including information indicating each image of a plurality of objects to be inspected captured by the imaging device 10 and associated with a classification label as teacher candidate image group information or teacher image group information.

[0050] The storage unit 20 also stores correct image group information, which is information on the correct image group for each class. The correct image group for a class is, for example, a typical image group, which is a group of images including multiple typical images of that class. A typical image for a class is an image that exhibits typical characteristics of that class.

[0051] The typical image group of a class is, for example, an image group including two or more images manually selected from among a plurality of images of an object to be inspected captured by the imaging device 10, but is not limited to such an example. For example, the typical image group of a class may be an image group including a plurality of images whose classification scores for that class are equal to or greater than a threshold, as output from a classification model. Note that the correct image group of a class used in the anomaly estimation process is not limited to a typical image group, and may include, for example, some images other than typical images.

[0052] <2.4.2. Processing Unit 21> The processing unit 21 realizes or executes the information processing functions and actions described below. As shown in Fig. 2, the processing unit 21 includes an image information acquisition unit 30, an inspection unit 31, an abnormality estimation unit 32, a classification score estimation unit 33, a classification accuracy determination unit 34, a classification accuracy information acquisition unit 35, a presentation unit 36, an operation detection unit 37, a change unit 38, and a learning unit 39. Note that the internal configuration of the processing unit 21 is not limited to the configuration shown in Fig. 2, and may be any other configuration that performs the information processing described below.

[0053] <2.4.2.1. Image information acquisition unit 30> The image information acquiring unit 30 acquires, for each object to be inspected, image information that indicates an image of the object to be inspected captured by the imaging device 10. The image information acquiring unit 30 stores the acquired image information in the storage unit 20.

[0054] <2.4.2.2. Inspection Unit 31> The inspection unit 31 performs an appearance inspection of the object to be inspected based on the image information acquired by the image information acquisition unit 30. The object to be inspected may be a product or workpiece produced on a factory production line, but may also be other objects. For example, the object to be inspected may be an agricultural product, a marine product, a forest product, or the like.

[0055] The inspection unit 31 performs an appearance inspection of the object to be inspected using a classification model that classifies images of the object to be inspected. The classification model is, for example, a model generated by machine learning in the learning unit 39 or the like, such as a convolutional neural network, but is not limited to this example and may be, for example, an autoencoder or other models.

[0056] The classification model is used, for example, for classification into the above-mentioned classes A, B, and C. Class A is, for example, a non-defective product, and classes B and C are, for example, different types of defective products. The number of classes classified by the classification model is not limited to three, A, B, and C, but may be four or more, or may be two. In the following description, it is assumed that the classes classified by the classification model are classes A, B, and C.

[0057] When image information acquired by the image information acquisition unit 30 is input, the classification model outputs classification scores for each of classes A, B, and C. The inspection unit 31 determines to which of classes A, B, and C the object to be inspected belongs, based on the classification scores for each of classes A, B, and C output from the classification model.

[0058] The inspection unit 31 classifies the object to be inspected into a class corresponding to the largest classification score among the classification scores of classes A, B, and C, for example, and determines the class into which the object to be inspected is classified as the appearance state of the object to be inspected.

[0059] Furthermore, the inspection unit 31 can also classify the object to be inspected into a class corresponding to the largest classification score that is equal to or greater than a threshold value among the classification scores of classes A, B, and C. In this case, for example, if there is no classification score that is equal to or greater than a threshold value among the classification scores of classes A, B, and C, the inspection unit 31 determines an unknown class that does not belong to any of classes A, B, and C as the appearance state of the object to be inspected.

[0060] An inspection object classified by the inspection unit 31 into class A is a non-defective product. An inspection object classified by the inspection unit 31 into class B is a defective product X, which is a first type of defective product. An inspection object classified by the inspection unit 31 into class C is a defective product Y, which is a second type of defective product.

[0061] The inspection unit 31 stores in the memory unit 20 teacher candidate information, which is information that associates the classes obtained by the above-mentioned visual inspection as classification labels with the images shown in the image information acquired by the image information acquisition unit 30.

[0062] <2.4.2.3. Abnormality estimation unit 32> The abnormality estimation unit 32 executes an abnormality estimation process to estimate an abnormality in each of a plurality of images each associated with a classification label as a training candidate image for a classification model.

[0063] The anomaly estimation unit 32 acquires, for example, teacher candidate image group information including information indicating a plurality of images each associated with a classification label as teacher candidate images for a classification model from the storage unit 20. The anomaly estimation unit 32 estimates an anomaly in each of the plurality of images indicated in the teacher candidate image group information acquired from the storage unit 20. Hereinafter, an image for which an anomaly is estimated by the anomaly estimation unit 32 may be referred to as a target image.

[0064] The abnormality estimation unit 32 estimates an abnormality in each of the multiple images based on, for example, the difference between the distribution of feature values ​​of the correct image group of the class classified by the classification model and the feature values ​​of each of the multiple target images. As described above, the correct image group of the class classified by the classification model is a typical image group that is an image group including multiple typical images as multiple images classified into that class.

[0065] Figure 3 is a diagram showing the relationship between the distribution of features of a group of typical images of a class and the distribution of features of a group of abnormal images of that class in a high-dimensional feature space. As shown in Figure 3, the distribution of features of a group of typical images of a class and the distribution of features of an abnormal image group classified into that class have different ranges in the feature space. Therefore, it is possible to estimate target images that have been erroneously classified into that class based on the difference in features from the distribution of features of the group of typical images.

[0066] The anomaly estimation unit 32 estimates an anomaly in the target image using, for example, PatchCore technology. First, the anomaly estimation unit 32 extracts the feature amounts of each correct image included in the correct image group. The feature amounts of the correct image are extracted using an anomaly detection model, which is a pre-trained deep learning model such as ResNet or EfficientNet. The deep learning model is, for example, a convolutional neural network, but is not limited to such an example. The feature amounts of the correct image are extracted for each patch, for example. The feature amounts of the correct image are represented by a high-dimensional feature vector.

[0067] The anomaly estimation unit 32 extracts features of each target image using the same extraction method as that used to extract features of the correct image. The anomaly estimation unit 32 then uses an anomaly detection model to compare, for each patch, a feature group including the features of each correct image included in the correct image group with the feature group of the target image, and calculates an anomaly score for each patch. The calculation of the anomaly score is performed using, for example, a nearest neighbor method, but is not limited to this example.

[0068] Then, the abnormality estimation unit 32 calculates the degree of abnormality of each pixel based on the degree of abnormality of each patch. For example, the abnormality estimation unit 32 calculates the degree of abnormality of each pixel in the target image by assigning the degree of abnormality of each patch to a corresponding position in the image.

[0069] The anomaly estimation unit 32 estimates an anomaly score as an index value indicating an anomaly in the target image based on the degree of anomaly of each pixel. The anomaly score includes a first anomaly score indicating the maximum value of the degree of anomaly in pixel units, and a second anomaly score indicating the proportion of areas where the degree of anomaly is equal to or greater than a threshold to the entire area of ​​the image. Note that the first anomaly score may be the average or median of the degree of anomaly of areas where the degree of anomaly is equal to or greater than a threshold.

[0070] In this way, the anomaly estimation unit 32 uses the anomaly detection model to estimate an anomaly score as the abnormal state of each of the multiple target images based on the difference between the distribution of features of the group of correct images of the class classified by the classification model and the feature values ​​of each of the multiple target images.

[0071] Alternatively, the anomaly estimation unit 32 may use an autoencoder as an anomaly detection model instead of PatchCore technology to estimate an anomaly in a target image. In this case, the anomaly estimation unit 32 trains the autoencoder using a group of correct images. In training the autoencoder, each correct image included in the group of correct images is used as an input image, and optimization is performed to minimize the difference between the input image and the output image. The autoencoder may be, for example, a model such as a convolutional neural network, but is not limited to such an example.

[0072] The anomaly estimation unit 32 inputs image information indicating a target image into a trained autoencoder, causing the autoencoder to compare the feature amounts of a group of correct images with those of the target image. The comparison result of the feature amounts of the group of correct images with those of the target image is expressed in the difference for each pixel between the image output from the autoencoder and the target image. The anomaly estimation unit 32 calculates the difference for each pixel between the image output from the autoencoder and the target image as the degree of anomaly.

[0073] The abnormality estimation unit 32 estimates an abnormality score for the target image as an index value indicating an abnormality in the target image based on the degree of abnormality of each pixel. The abnormality score includes a first abnormality score indicating the maximum value of the degree of abnormality for each pixel, and a second abnormality score indicating the proportion of areas where the degree of abnormality is equal to or greater than a threshold to the entire area of ​​the image. Note that the first abnormality score may be, for example, the average value or median value of the degree of abnormality of areas where the degree of abnormality is equal to or greater than a threshold.

[0074] In this way, the anomaly estimation unit 32 can use an autoencoder as an anomaly detection model to estimate an anomaly score as an abnormal state of the target image based on the difference between the feature values ​​of the correct image group of classes classified by the classification model and the feature values ​​of the target image.

[0075] The abnormality estimation unit 32 can also estimate whether or not the target image has an abnormality based on the estimated abnormality score. For example, the abnormality estimation unit 32 estimates that the target image has an abnormality when the first abnormality score is equal to or greater than a threshold, when the second abnormality score is equal to or greater than a threshold, or when the score obtained by weighting the first abnormality score and the second abnormality score is equal to or greater than a threshold. An abnormality in the target image is an abnormality in the appearance of the object to be inspected shown in the target image.

[0076] The estimation of an abnormality in the target image by the anomaly estimation unit 32 is not limited to the above-described example. For example, the anomaly estimation unit 32 can also estimate an abnormality in the target image by pattern matching processing. The anomaly estimation unit 32 has, for example, a correct answer image for each class, extracts features of the target image, and estimates that the image in the class has the closest similarity to the features of the correct answer image for each class. The image features are extracted by, for example, SIFT (Scale-Invariant Feature Transform) or SURF (Speeded Up Robust Feature), but are not limited to these examples.

[0077] <2.4.2.4. Classification score estimation unit 33> The classification score estimation unit 33 estimates the classification score of each of the multiple images using the classification model.

[0078] For example, the classification score estimation unit 33 inputs the image information acquired by the image information acquisition unit 30 into the classification model, and estimates the classification score of each class output from the classification model as the classification score of the image. Also, the classification score estimation unit 33 inputs information indicating the teacher candidate image stored in the storage unit 20 into the classification model, and estimates the classification score of each class output from the classification model as the classification score of the teacher candidate image.

[0079] <2.4.2.5.Classification accuracy determination unit 34> The classification accuracy determination unit 34 determines the value of an evaluation index related to the classification accuracy of the classification model. The classification accuracy determination unit 34 determines the value of an evaluation index related to the classification accuracy of the classification model, for example, based on the result of the visual inspection of the inspected object by the inspector O and the classification result by the classification model in the inspection unit 31.

[0080] The visual inspection of the object to be inspected by the inspector O is performed, for example, based on an image represented by image information acquired by the image information acquisition unit 30. For example, the classification results based on the classification model in the inspection unit 31 are presented to the inspector O by the presentation unit 36, and the inspector O can input the results of the visual inspection of the object to be inspected by operating the operation device 12 in response to the classification results based on the classification model, as will be described later.

[0081] Evaluation indices relating to the classification accuracy of a classification model include, for example, the false negative rate, false positive rate, and recall rate. The false negative rate is the proportion of images that are erroneously classified by the classification model as a class other than the target class, out of multiple images that belong to the target class, which is the class to be evaluated, and is also called the oversight rate. When the classification model classifies an image into one of the above-mentioned classes A, B, and C, for example, the target class is class B or C, which indicates a defective product. Hereinafter, the false negative rate may be referred to as the oversight rate.

[0082] The false positive rate is the proportion of images that are erroneously classified into the target class by the classification model among multiple images that do not belong to the target class, and is also called the overdetection rate. When the classification model classifies an image into one of the above-mentioned classes A, B, or C, the target class is, for example, class A, which indicates a non-defective product. Hereinafter, the false positive rate may be referred to as the overdetection rate.

[0083] The recall rate is the percentage of images that the classification model correctly classifies into a target class among multiple images belonging to that class. If the classification model classifies images into one of the above-mentioned classes A, B, or C, the target class is, for example, class B or C, which indicates defective products.

[0084] The evaluation index for the classification accuracy of a classification model is not limited to the false negative rate, false positive rate, and recall rate, and may be expressed as, for example, accuracy, F1 score, or other evaluation index. Accuracy is the proportion of images classified into the correct class among multiple images classified by the classification model. The F1 score is a value calculated from the accuracy and recall rate.

[0085] When the value of the evaluation index relating to the classification accuracy determined by the classification accuracy determination unit 34 is calculated for each class, it may be, for example, the average value or median value of the evaluation index for each class.

[0086] The classification accuracy determination unit 34 performs the above-mentioned determination at predetermined intervals, and stores information indicating the results of the determination in the storage unit 20 as classification accuracy information.

[0087] <2.4.2.6.Classification accuracy information acquisition unit 35> The classification accuracy information acquisition unit 35 acquires classification accuracy information, which is information relating to the classification accuracy of the classification model.

[0088] The classification accuracy information acquisition unit 35 acquires, as classification accuracy information, information indicating the value of the evaluation index determined by the classification accuracy determination unit 34 from the storage unit 20. The classification accuracy information acquisition unit 35 can also acquire classification accuracy information from an external device.

[0089] <2.4.2.7. Presentation part 36> The presentation unit 36 ​​presents various pieces of information to the inspector O by displaying them on the display device 11. For example, the presentation unit 36 ​​displays an inspection screen on the display device 11, the inspection screen including an image represented by the image information acquired by the image information acquisition unit 30 and an inspection result image including the inspection result by the inspection unit 31. The inspection screen also includes a GUI (Graphical User Interface) that can be operated by the inspector O.

[0090] 4 is a diagram showing an example of an inspection screen displayed on the display device 11 by the presentation unit 36 ​​in the processing unit 21 of the image inspection device 13 according to the embodiment. As shown in FIG. 4, the inspection screen 50 displayed on the display device 11 by the presentation unit 36 ​​includes an inspection object area 51 in which an image of the object inspected by the inspection unit 31 is arranged, and an inspection result area 52 in which the inspection results by the inspection unit 31 are displayed. Furthermore, the inspection screen 50 includes an operation area 53 including a GUI operated by the inspector O.

[0091] Information indicating that the inspection result of the inspection unit 31 on the inspection object is a pass product is arranged in the inspection object area 51 shown in Fig. 4. Also, an OK button 531, a pass product button 532, a defective product X button 533, and a defective product Y button 534 are arranged in the operation area 53 shown in Fig. 4.

[0092] The OK button 531 is a button that is selected by the inspector O operating the operation device 12 when the inspector O determines that the inspection result by the inspection unit 31 shown in the inspection result area 52 is correct.

[0093] The good product button 532, the defective product X button 533, and the defective product Y button 534 are buttons that are selected by the inspector O operating the operation device 12 when the inspector O determines that the inspection results by the inspection unit 31 shown in the inspection result area 52 are incorrect.

[0094] For example, when an inspected object that is a defective product X is determined by the inspection unit 31 to be a good product or a defective product Y, the inspector O selects the defective product X button 533. When an inspected object that is a defective product Y is determined by the inspection unit 31 to be a good product or a defective product X, the inspector O selects the defective product Y button 534. When an inspected object that is a good product is determined by the inspection unit 31 to be a defective product X or a defective product Y, the inspector O selects the good product button 532.

[0095] 2 presents at least some of the teacher candidate images among the plurality of teacher candidate images based on the estimation result by the anomaly estimation unit 32. For example, the presentation unit 36 ​​presents at least some of the teacher candidate images among the plurality of teacher candidate images based on at least one of the anomaly score estimated by the anomaly estimation unit 32 and the classification score estimated by the classification score estimation unit 33.

[0096] The presentation unit 36 ​​includes a sorting processing unit 40 that sorts multiple teacher candidate images based on the abnormality score estimated by the abnormality estimation unit 32, and a presentation processing unit 41 that presents at least some of the teacher candidate images based on the sorting results by the sorting processing unit 40.

[0097] The sorting processing unit 40 has multiple sorting modes, including a first sorting mode and a second sorting mode. The first sorting mode is a mode in which the sorting method for multiple teacher candidate images is switched based on the classification accuracy information acquired by the classification accuracy information acquisition unit 35. The second sorting mode is a mode in which the multiple teacher candidate images are sorted based on the evaluation score based on the classification accuracy information acquired by the classification accuracy information acquisition unit 35. The inspector O or the administrator of the image inspection device 13 can select a sorting mode to be executed by the sorting processing unit 40 from the multiple sorting modes. The sorting processing unit 40 executes the selected sorting mode.

[0098] First, the first sorting mode will be described. In the first sorting mode, the sorting processing unit 40 switches the sorting method of the multiple teacher candidate images from sorting based on the anomaly score to sorting based on the classification score, based on the classification accuracy information acquired by the classification accuracy information acquisition unit 35. As described above, the anomaly score is a score estimated by the anomaly estimation unit 32, and the classification score is a score estimated by the classification score estimation unit 33.

[0099] For example, when the value of the evaluation index indicated by the classification accuracy information is the oversight rate, and when the oversight rate indicated by the classification accuracy information is greater than a threshold, the sorting processing unit 40 sorts multiple teacher candidate images that are teacher candidate images by sorting based on the anomaly score.

[0100] Sorting based on the abnormality score is, for example, sorting based on the magnitude of the first abnormality score, sorting based on the magnitude of the second abnormality score, or sorting based on the magnitude of the result of normalizing the first abnormality score and the second abnormality score and then weighting and adding them.

[0101] Furthermore, when the overlooked rate indicated by the classification accuracy information is equal to or lower than a threshold, the sorting unit 40 sorts the multiple teacher candidate images that are teacher candidate images based on the classification score. The threshold is, for example, zero or a value greater than zero. Note that the value of the evaluation index used to switch the sorting method is not limited to the overlooked rate, but may be the overdetection rate or another evaluation index value.

[0102] Next, the second sorting mode will be described. In the second sorting mode, the sorting processing unit 40 sorts a plurality of images based on the evaluation scores calculated based on the anomaly scores estimated by the anomaly estimation unit 32 and the classification scores estimated by the classification score estimation unit 33.

[0103] For example, the sorting processing unit 40 calculates an evaluation score for each of the multiple images by weighted addition of multiple scores including a first score obtained by normalizing the first abnormality score, a second score obtained by normalizing the second abnormality score, and a third score obtained by normalizing the classification score.

[0104] The sorting unit 40 calculates an evaluation score for each of the images by weighting and adding the scores using weights according to the classification accuracy information. For example, the sorting unit 40 calculates the evaluation score using the following formula (1).

[0105]

number

[0106] In the above formula (1), f[n] represents the evaluation score, and m represents the number of evaluation axes to be weighted and added. The number of evaluation axes to be weighted and added is 3 when the targets of weighting and addition are the first score, the second score, and the third score. Also, in the above formula (1), ω k [n] indicates the weight, ω1[n]~ω m The total value of [n] is 1. In addition, in the above formula (1), x kis a score indicating a normalized evaluation value, such as the first score, second score, third score, etc. k is not limited to the first score, second score, and third score.

[0107] For example, the sorting unit 40 increases the weight of the third score and decreases the weights of the first score and the second score as the value of the evaluation index relating to the classification accuracy indicated by the classification accuracy information improves.

[0108] FIG. 5 is a diagram showing an example of the relationship between the weight of each score by the sorting processing unit 40 in the processing unit 21 of the image inspection device 13 according to the embodiment and the value of the evaluation index. In FIG. 5, the evaluation index is, for example, the overlooked detection rate, the overdetection rate, the recall rate, the accuracy rate, or the F1 score. When the evaluation index is the overlooked detection rate or the overdetection rate, the value of the evaluation index decreases in the direction of the arrow on the horizontal axis. When the evaluation index is the recall rate, the accuracy rate, or the F1 score, the value of the evaluation index increases in the direction of the arrow on the horizontal axis. Furthermore, the magnitude of the weight increases in the direction of the arrow on the vertical axis.

[0109] When the evaluation index for classification accuracy is the overlooked rate or the overdetected rate, the sorting unit 40 increases the weight of the third score and decreases the weights of the first score and the second score as the overlooked rate or the overdetected rate decreases.

[0110] In addition, when the evaluation index is recall, accuracy, or F1 score, the sorting processing unit 40 increases the weight of the third score and decreases the weights of the first score and the second score as the recall, accuracy, or F1 score increases.

[0111] The presentation processing unit 41 displays various pieces of information on the display device 11, thereby presenting the various pieces of information to the inspector O. For example, the presentation processing unit 41 displays an examination screen 50 shown in FIG. 4 on the display device 11.

[0112] The presentation processor 41 also presents information indicating weights according to the classification accuracy information for each of the first score, the second score, and the third score, and information relating to the classification accuracy indicated by the classification accuracy information. The presentation processor 41 also presents information indicating the information relating to the classification accuracy indicated by the classification accuracy information in chronological order.

[0113] For example, the presentation processing unit 41 presents to the inspector O information showing the information on the classification accuracy indicated by the classification accuracy information in chronological order by displaying on the display device 11 a management screen including information showing the values ​​of the evaluation index, which is information on the classification accuracy indicated by the classification accuracy information, in chronological order.

[0114] 6 is a diagram showing an example of a management screen displayed on the display device 11 by the presentation processing unit 41 in the processing unit 21 of the image inspection device 13 according to the embodiment. The management screen 60 shown in FIG. 6 includes an evaluation index selection box 61, display buttons 62 and 63, a comparison button 64, and a classification accuracy information display area 65.

[0115] The evaluation index selection box 61 is a combo box for selecting the type of evaluation index whose value is displayed in chronological order in the classification accuracy information display area 65. The display buttons 62 and 63 are buttons for displaying the inspection results by the inspection unit 31 for a specific period on the display device 11. The comparison button 64 is a button for displaying the inspection results by the inspection unit 31 for multiple specific periods in a comparable manner on the display device 11.

[0116] Assume that inspector O operates the operation device 12 to operate the evaluation index selection box 61, and selects from multiple types of evaluation indexes an evaluation index whose value is to be displayed in the classification accuracy information display area 65. In this case, the presentation processing unit 41 displays the evaluation index selected by inspector O at the selected position in the evaluation index selection box 61, and places information in the classification accuracy information display area 65 that shows the values ​​of the evaluation index selected by inspector O in chronological order.

[0117] Also, assume that inspector O selects either display button 62 or 63 by operating operation device 12. In this case, presentation processing unit 41 causes display device 11 to display the inspection results by inspection unit 31 for a specific period corresponding to the button selected by inspector O.

[0118] Also, assume that inspector O selects comparison button 64 by operating operation device 12. In this case, presentation processing unit 41 causes display device 11 to display, in a comparable manner, the inspection results by inspection unit 31 for a plurality of specific periods corresponding to the button selected by inspector O.

[0119] Also, assume that inspector O operates the operation device 12 to select a circle plotted as an evaluation index value in the classification accuracy information display area 65. The circle is selected by, for example, a click operation or a mouse-over operation. In this case, the presentation processing unit 41 causes the display device 11 to display information indicating weights according to the classification accuracy information for each of the first score, second score, and third score for the period corresponding to the selected circle.

[0120] In this way, when the management screen 60 is operated by the inspector O operating the operation device 12, the presentation processing unit 41 can perform display processing in accordance with the operation by the inspector O on the operation device 12.

[0121] Fig. 7 is a diagram showing an example of an additional information screen displayed on the display device 11 by the presentation unit 36 ​​in the processing unit 21 of the image inspection device 13 according to the embodiment. The additional information screen 70A shown in Fig. 7 is a screen displayed on the display device 11 by the presentation processing unit 41 when the display button 63 shown in Fig. 6 is selected.

[0122] 7, multiple pieces of work information for January 21 are classified and displayed as additional information. The additional information includes information indicating teacher candidate images that can be used for additional learning of the classification model. Each piece of work information includes information on the character string "work information" and information indicating the teacher candidate image.

[0123] Furthermore, on the additional information screen 70A, multiple pieces of work information for class A are displayed sorted in descending order from the work information for the teacher candidate image with the highest evaluation score to the work information for the teacher candidate image with the lowest evaluation score. On the additional information screen 70A, two pieces of work information for class C are displayed sorted in descending order from the highest evaluation score for the teacher candidate image, and no work information is classified into class B.

[0124] On the additional information screen 70A, the work information is arranged for each class in a position sorted according to the evaluation score for the teacher candidate image, but this is not limiting, and for example, the work information may be arranged for each class in the order of appearance inspection by the inspection unit 31. In this case, the presentation processing unit 41 causes the display device 11 to display the additional information screen 70A, which shows, in a highlighted state, work information including a teacher candidate image that satisfies a predetermined condition.

[0125] The predetermined condition is, for example, but not limited to, a condition that the evaluation score is equal to or less than a threshold value or a condition that the evaluation score is within the bottom m ranks, where m is an integer of 2 or greater.

[0126] When work information is selected by the inspector O operating the operation device 12, the presentation processing unit 41 causes the display device 11 to enlarge and display the teacher candidate image corresponding to the selected work information. Based on the teacher candidate image enlarged and displayed on the display device 11, it is relatively easy to determine whether the inspection result by the inspection unit 31 is correct or not.

[0127] Furthermore, if the teacher candidate image corresponding to the selected work information has a classification label change history, the presentation processing unit 41 can also display the classification label change history on the display device 11 in addition to the teacher candidate image.

[0128] 7 displays all work information inspected by the inspection unit 31 on January 21st, but is not limited to this example. For example, the presentation processing unit 41 extracts, for each class, work information including teacher candidate images whose evaluation scores satisfy the above-mentioned predetermined conditions from among the multiple pieces of work information on January 21st, and causes the display device 11 to display the additional information screen 70A including the extracted work information for each class.

[0129] The inspector O can perform a change operation to move one or more pieces of work information that have been determined to be incorrectly classified into the correct class to the correct class by operating the operation device 12. On the additional information screen 70B shown in Fig. 7, a change operation performed by operating the operation device 12 moves one piece of work information classified into class A to class B. This change operation is a change operation of the classification label.

[0130] Fig. 8 is a diagram showing another example of an additional information screen displayed on the display device 11 by the presentation unit 36 ​​in the processing unit 21 of the image inspection device 13 according to the embodiment. The additional information screen 80 shown in Fig. 8 is a screen displayed on the display device 11 by the presentation processing unit 41 when the comparison button 64 shown in Fig. 6 is selected.

[0131] 8, multiple pieces of work information for each of January 20th and January 21st are classified and displayed as additional information. The additional information for January 21st is not included in the additional information for January 20th. This allows inspector O to easily determine the classification status for each period.

[0132] On the additional information screen 80 shown in Fig. 8, similarly to the additional information screen 70A shown in Fig. 7, when work information is selected by the inspector O operating the operation device 12, a teacher candidate image corresponding to the selected work information can be enlarged and displayed on the display device 11. Furthermore, the inspector O can operate the operation device 12 to perform a change operation on the additional information screen 80 to move one or more pieces of work information that have been determined to be incorrectly classified into the correct class to the correct class.

[0133] <2.4.2.8. Operation detection unit 37> The operation detection unit 37 detects various operations performed by the inspector O using the operation device 12. For example, while the inspection screen 50 is displayed on the display device 11, the operation detection unit 37 detects the selection by the inspector O of any of the OK button 531, the non-defective button 532, the defective X button 533, and the defective Y button 534 included in the inspection screen 50.

[0134] In addition, the operation detection unit 37 detects an operation to change the classification label for one or more teacher candidate images presented by the presentation unit 36 ​​when the multiple teacher candidate images presented by the presentation unit 36 ​​are displayed on the display device 11.

[0135] <2.4.2.9. Change section 38> The change unit 38 changes the classification labels of teacher candidate images for which a classification label change operation has been performed among at least some of the multiple teacher candidate images presented by the presentation unit 36. The classification label change operation is detected by the operation detection unit 37, as described above.

[0136] For example, suppose that work information classified as class A is moved to class B by an operation by inspector O, as shown in additional information screen 70B in Fig. 7. In this case, the change unit 38 changes the classification label associated with the teacher candidate image corresponding to the work information moved to class B from class A to class B.

[0137] <2.4.2.10. Learning Section 39> The learning unit 39 learns the classification model using a plurality of teacher images. Each of the plurality of teacher images is associated with a classification label, and the learning unit 39 learns the classification model using learning information including the teacher image and the classification label for each teacher image.

[0138] The learning unit 39 learns the classification model through an optimization process using a plurality of teacher candidate images as a plurality of teacher images. The learning unit 39 learns the classification model through an optimization process using a plurality of teacher candidate images including teacher candidate images whose classification labels have been changed by the changing unit 38, for example.

[0139] <3. Example of processing procedure of image inspection device 13> Next, a procedure for information processing by the processing unit 21 of the image inspection device 13 according to the embodiment will be described. Fig. 9 is a flowchart showing an example of information processing by the processing unit 21 of the image inspection device 13 according to the embodiment.

[0140] 9, the processing unit 21 of the image inspection device 13 determines whether or not it is time to perform a visual inspection (step S10). The visual inspection timing is the timing when a new object to be inspected is imaged by the imaging device 10, but may also be the timing when an inspection start operation is performed by the inspector O on the operation device 12.

[0141] When it is determined that it is time to perform a visual inspection (step S10: Yes), processing unit 21 executes a visual inspection process for the object to be inspected (step S11). The process of step S11 is the process of steps S20 to S24 shown in FIG. 10, and will be described in detail later.

[0142] When the processing of step S11 is completed or when it is determined that it is not time for visual inspection (step S10: No), the processing unit 21 determines whether it is time for classification correction (step S12). The classification correction timing is, for example, the timing when the inspector O performs a classification correction operation on the operation device 12, but is not limited to this example.

[0143] When it is determined that the timing for correcting the classification has arrived (step S12: Yes), the processing unit 21 executes a classification correction process (step S13). The classification correction process of step S13 is the process of steps S30 to S36 shown in Fig. 11, and will be described in detail later.

[0144] When the processing of step S13 is completed or when it is determined that the timing for correcting the classification has not come (step S12: No), the processing unit 21 determines whether the timing for learning has come (step S14). The timing for learning is, for example, the timing when the inspector O performs a learning start operation on the operation device 12, but is not limited to this example.

[0145] When it is determined that it is time to learn (step S14: Yes), the processing unit 21 performs learning of the classification model using the teacher candidate images associated with the classification labels as teacher images (step S15).

[0146] When the processing of step S15 is completed or when it is determined that the learning timing has not come (step S14: No), the processing unit 21 determines whether the operation end timing has come (step S16). The processing unit 21 determines that the operation end timing has come when, for example, the power of the image inspection device 13 is turned off.

[0147] If the processing unit 21 determines that the operation end time has not yet arrived (step S16: No), it proceeds to step S10, and if it determines that the operation end time has arrived (step S16: Yes), it terminates the processing shown in Figure 9.

[0148] 10 is a flowchart showing an example of the appearance inspection process by the processing unit 21 of the image inspection device 13 according to the embodiment. As shown in Fig. 10, the processing unit 21 acquires image information, which is information indicating an image of an object to be inspected, from the imaging device 10 (step S20).

[0149] Next, the processing unit 21 uses the classification model to classify the image indicated by the image information acquired in step S20 (step S21), and then the processing unit 21 displays the classification result of step S21 on the display device 11 (step S22).

[0150] Next, the processing unit 21 accepts an operation from the inspector O (step S23). The processing unit 21 determines the image indicated by the image information acquired in step S20 as a teacher candidate image, and stores the teacher candidate image as teacher candidate image group information, which is information in which the teacher candidate image is associated with a classification label according to the operation accepted in step S23, in the storage unit 20 (step S24). Then, when the processing of step S24 ends, the processing unit 21 ends the processing shown in FIG. 10.

[0151] 11 is a flowchart showing an example of classification correction processing by the processing unit 21 of the image inspection device 13 according to the embodiment. As shown in Fig. 11, the processing unit 21 acquires teacher candidate image group information from the storage unit 20, and estimates abnormalities in each of a plurality of images associated with classification labels as teacher candidate images for the classification model (step S30).

[0152] Next, the processing unit 21 acquires classification accuracy information, which is information about the classification accuracy of the classification model, from the storage unit 20 or an external device (step S31), and executes an evaluation score determination process based on the acquired classification accuracy information (step S32). The evaluation score determination process of step S32 is the process of steps S40 to S45 shown in Fig. 12, and will be described in detail later.

[0153] Next, the processing unit 21 executes a sorting process based on the evaluation scores determined in step S32 (step S33). Then, the processing unit 21 displays at least some of the teacher candidate images among the plurality of images on the display device 11 based on the sorting result of step S33 (step S34).

[0154] Next, the processing unit 21 determines whether or not the inspector O has performed a change operation on the classification labels (step S35). If the processing unit 21 determines that a change operation has been performed (step S35: Yes), it changes the classification labels of one or more teacher candidate images that have been subject to the change operation (step S36).

[0155] When the processing of step S36 is completed or when it is determined that no change operation has been performed (step S35: No), the processing unit 21 ends the processing shown in FIG.

[0156] 12 is a flowchart showing an example of an evaluation score determination process by the processing unit 21 of the image inspection device 13 according to the embodiment. As shown in FIG. 12, the processing unit 21 determines whether the sorting mode is set to the first sorting mode (step S40).

[0157] When it is determined that the first sorting mode is set (step S40: Yes), the processing unit 21 determines whether the value of the evaluation index satisfies a predetermined condition (step S41). When it is determined that the value of the evaluation index satisfies the predetermined condition (step S41: Yes), the processing unit 21 determines the evaluation score by anomaly estimation (step S42). For example, when the value of the evaluation index is a miss-detection rate, the miss-detection rate is greater than a threshold.

[0158] In step S42, the processing unit 21 determines, for example, the first abnormality score, the second abnormality score, or a score obtained by normalizing the first abnormality score and the second abnormality score and then performing a weighted addition, as the evaluation score.

[0159] Furthermore, when the processing unit 21 determines that the value of the evaluation index does not satisfy the predetermined condition (step S41: No), it determines the evaluation score in the classification model (step S43). In step S42, the processing unit 21 determines the classification score of the classification model for the teacher candidate image as the evaluation score. The classification score of the classification model for the teacher candidate image is the classification score output from the classification model when information indicating the teacher candidate image is input to the classification model.

[0160] If it is determined that the first sorting mode is not set (step S40: No), the processing unit 21 calculates the weight of each evaluation axis (step S44) and calculates the evaluation score using the calculated weight (step S45). The weight of each evaluation axis is, for example, the weight of the first score, the weight of the second score, and the weight of the third score described above.

[0161] When the processing of step S42 is completed, when the processing of step S43 is completed, or when the processing of step S45 is completed, the processing unit 21 ends the processing shown in FIG.

[0162] <4. Hardware Configuration> FIG. 13 is a diagram showing an example of the hardware configuration of the processing unit 21 in the image inspection device 13 according to the embodiment.

[0163] 13, the processing unit 21 has a processor 90, a memory 91, an input / output interface (I / F) 92, and a media I / F (I / F) 93. The processor 90, the memory 91, the input / output interface 92, and the media interface 93 are connected by a bus 95.

[0164] The processor 90 includes, for example, one or more of a central processing unit (CPU), a micro processing unit (MPU), a graphics processing unit (GPU), and a system large scale integration (LSI). The memory 91 is a random access memory (RAM) such as a static random access memory (SRAM) or a dynamic random access memory (DRAM), or a semiconductor memory element.

[0165] The processor 90 transmits and receives information to and from output devices such as the display device 11, and transmits and receives information to and from input devices such as the imaging device 10 and the operation device 12, via the input / output interface 92. The processor 90 acquires data from the input devices via the input / output interface 92. The processor 90 also outputs generated data to the output devices via the input / output interface 92.

[0166] The media interface 93 reads a program or data stored in a recording medium 94 and provides the read data or program (an example of a classification correction program) to the processor 90 via the memory 91. The processor 90 loads the program from the recording medium 94 onto the memory 91 via the media interface 93 and executes the loaded program. The recording medium 94 is, for example, an optical recording medium such as a DVD (Digital Versatile Disc) or a PD (Phase Change Rewritable Disc), or a semiconductor memory.

[0167] The processor 90 executes programs using the memory 91 or the like as a working area, thereby realizing the functions of the image information acquisition unit 30, the inspection unit 31, the abnormality estimation unit 32, the classification score estimation unit 33, the classification accuracy determination unit 34, the classification accuracy information acquisition unit 35, the presentation unit 36, the operation detection unit 37, the change unit 38, and the learning unit 39.

[0168] The processing unit 21 may be partially or entirely realized by an integrated circuit such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or a digital signal processor (DSP).

[0169] Although an embodiment of the present invention has been described above, this embodiment is presented as an example and is not intended to limit the scope of the invention. This embodiment can be embodied in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included within the scope and spirit of the invention, as well as within the scope of the invention and its equivalents as set forth in the claims.

[0170] The present technology can be configured as follows. (1) An image inspection device that performs image inspection using a classification model that classifies images of an object to be inspected, an anomaly estimation unit that estimates an anomaly in each of a plurality of images associated with a classification label as candidate teacher images for the classification model; a presentation unit that presents at least some of the images based on an estimation result by the abnormality estimation unit; an alteration unit that alters the classification label of an image for which an alteration operation for the classification label has been performed, among the at least some of the images presented by the presentation unit. (2) The image inspection device described in (1) is provided with a learning unit that trains the classification model using the plurality of images including the image whose classification label has been changed by the change unit. (3) The abnormality estimation unit An image inspection device according to (1) or (2), which estimates abnormalities in each of the plurality of images based on the difference between the feature values ​​of a group of correct images of a class classified by the classification model and the feature values ​​of each of the plurality of images. (4) The abnormality estimation unit estimating an anomaly score for each of the plurality of images using an anomaly detection model; The presentation unit a sorting processing unit that sorts the plurality of images based on the anomaly scores estimated by the anomaly estimation unit; The image inspection device according to any one of (1) to (3), further comprising: a presentation processing unit that presents at least some of the images based on a result of sorting by the sort processing unit. (5) a classification score estimation unit that estimates a classification score for each of the plurality of images using the classification model; a classification accuracy information acquisition unit that acquires classification accuracy information that is information about classification accuracy by the classification model, The sorting unit The image inspection device according to (4), wherein the sorting method of the plurality of images is switched from sorting based on the anomaly scores estimated by the anomaly estimation unit to sorting based on the classification scores estimated by the classification score estimation unit, based on the classification accuracy information acquired by the classification accuracy information acquisition unit. (6) A classification score estimation unit that estimates a classification score for each of the plurality of images using the classification model, The sorting unit The image inspection device according to (4), wherein the plurality of images are sorted based on an evaluation score calculated based on the anomaly score estimated by the anomaly estimation unit and the classification score estimated by the classification score estimation unit. (7) The anomaly score for each of the plurality of images is a first anomaly score indicating the maximum value of the degree of anomaly in pixel units, and a second anomaly score indicating the ratio of an area where the degree of anomaly is equal to or greater than a threshold to an entire area of ​​the image, The sorting unit The image inspection device described in (6) calculates the evaluation score for each of the multiple images by weighted addition of multiple scores including a first score normalized from the first abnormality score, a second score normalized from the second abnormality score, and a third score normalized from the classification score. (8) A classification accuracy information acquisition unit that acquires classification accuracy information, which is information about the classification accuracy of the classification model, The sorting unit The image inspection device according to (7), wherein the evaluation score for each of the plurality of images is calculated by weighted addition of the plurality of scores using weights according to the classification accuracy information. (9) The presentation processing unit An image inspection device as described in (8), which presents information indicating weights of the first score, the second score, and the third score according to the classification accuracy information, and information regarding the classification accuracy indicated by the classification accuracy information. (10) The presentation processing unit The image inspection device according to (9), which presents information showing information relating to classification accuracy indicated by the classification accuracy information in chronological order. (11) A classification score estimation unit is provided that estimates a classification score for each of the plurality of images using the classification model; The abnormality estimation unit estimating an anomaly score for each of the plurality of images using an anomaly detection model; The presentation unit An image inspection device according to any one of (1) to (3), which presents at least some of the images among the plurality of images based on an evaluation score calculated based on at least one of the abnormality score estimated by the abnormality estimation unit and the classification score estimated by the classification score estimation unit. (12) A computer-implemented determination method, an anomaly estimation step of estimating an anomaly in each of a plurality of images associated with a classification label as candidate teacher images for a classification model that classifies images of an object to be inspected; a presentation step of presenting at least some of the images among the plurality of images based on an estimation result obtained by the abnormality estimation step; a changing step of changing the classification label of an image for which a change operation of the classification label has been performed, among the at least some of the images presented by the presenting step. (13) an anomaly estimation step for estimating an anomaly in each of a plurality of images associated with classification labels as candidate images for training a classification model for classifying images of an object to be inspected; a presentation step of presenting at least some of the images among the plurality of images based on an estimation result from the abnormality estimation step; a modification procedure for modifying the classification label of an image for which a modification operation of the classification label has been performed, among the at least some of the images presented by the presentation procedure;

[0171] The disclosed embodiments should be considered in all respects as illustrative and not restrictive. Indeed, the above-described embodiments may be embodied in various forms. Furthermore, the above-described embodiments may be omitted, substituted, or modified in various forms without departing from the scope and spirit of the appended claims. [Explanation of symbols]

[0172] 10. Imaging device 11 Display device 12 Operating device 13 Image inspection equipment 20 Memory section 21 Processing section 30 Image information acquisition unit 31 Inspection Department 32 Abnormality estimation part 33 Classification score estimation part 34 Classification accuracy determination unit 35 Classification accuracy information acquisition unit 36 Presentation part 37 Operation detection unit 38 Changes 39 Learning Department 40 Sorting processing section 41 Presentation processing unit 100 Inspection Systems

Claims

1. 1. An image inspection apparatus that performs image inspection using a classification model that classifies images of an object to be inspected, an anomaly estimation unit that estimates an anomaly in each of a plurality of images associated with a classification label as candidate teacher images for the classification model; a presentation unit that presents at least some of the images based on an estimation result by the abnormality estimation unit; an alteration unit that alters the classification label of an image for which an alteration operation for the classification label has been performed, among the at least some of the images presented by the presentation unit.

2. The image inspection device according to claim 1 , further comprising a learning unit that learns the classification model using the plurality of images including the image whose classification label has been changed by the change unit.

3. The abnormality estimation unit 3. The image inspection device according to claim 1, wherein an abnormality in each of the plurality of images is estimated based on a difference between a feature amount of a group of correct images of a class classified by the classification model and a feature amount of each of the plurality of images.

4. The abnormality estimation unit estimating an anomaly score for each of the plurality of images using an anomaly detection model; The presentation unit a sorting processing unit that sorts the plurality of images based on the anomaly scores estimated by the anomaly estimation unit; The image inspection device according to claim 1 , further comprising: a presentation processing unit that presents at least some of the images based on a result of sorting by the sort processing unit.

5. a classification score estimation unit that estimates a classification score for each of the plurality of images using the classification model; a classification accuracy information acquisition unit that acquires classification accuracy information that is information about classification accuracy by the classification model, The sorting processing unit 5. The image inspection device according to claim 4, wherein a sorting method of the plurality of images is switched from sorting based on the anomaly scores estimated by the anomaly estimation unit to sorting based on the classification scores estimated by the classification score estimation unit, based on the classification accuracy information acquired by the classification accuracy information acquisition unit.

6. a classification score estimation unit that estimates a classification score for each of the plurality of images using the classification model; The sorting processing unit 5. The image inspection device according to claim 4, wherein the plurality of images are sorted based on an evaluation score calculated based on the anomaly score estimated by the anomaly estimation unit and the classification score estimated by the classification score estimation unit.

7. The anomaly score for each of the plurality of images is: a first anomaly score indicating a maximum value of the degree of anomaly in pixel units, and a second anomaly score indicating a ratio of an area in which the degree of anomaly is equal to or greater than a threshold to an entire area of ​​the image, The sorting processing unit 7. The image inspection device according to claim 6, wherein the evaluation score for each of the plurality of images is calculated by weighted addition of a plurality of scores including a first score obtained by normalizing the first anomaly score, a second score obtained by normalizing the second anomaly score, and a third score obtained by normalizing the classification score.

8. a classification accuracy information acquisition unit that acquires classification accuracy information that is information about the classification accuracy of the classification model; The sorting processing unit The image inspection device according to claim 7 , wherein the evaluation score for each of the plurality of images is calculated by weighted addition of the plurality of scores using weights according to the classification accuracy information.

9. The presentation processing unit 9. The image inspection device according to claim 8, further comprising: information indicating weights of the first score, the second score, and the third score according to the classification accuracy information; and information regarding the classification accuracy indicated by the classification accuracy information.

10. The presentation processing unit The image inspection device according to claim 9 , wherein information indicating information relating to the classification accuracy indicated by the classification accuracy information in chronological order is presented.

11. a classification score estimation unit that estimates a classification score for each of the plurality of images using the classification model; The abnormality estimation unit estimating an anomaly score for each of the plurality of images using an anomaly detection model; The presentation unit 3. The image inspection device according to claim 1, wherein at least some of the images among the plurality of images are presented based on an evaluation score calculated based on at least one of the anomaly score estimated by the anomaly estimation unit and the classification score estimated by the classification score estimation unit.

12. 1. A computer-implemented classification correction method comprising: an anomaly estimation step of estimating an anomaly in each of a plurality of images associated with classification labels as candidate teacher images for a classification model that classifies images of the object to be inspected; a presentation step of presenting at least some of the images among the plurality of images based on an estimation result obtained by the abnormality estimation step; a changing step of changing the classification label of an image for which a change operation of the classification label has been performed, among the at least some of the images presented by the presenting step.

13. an anomaly estimation procedure for estimating anomalies in each of a plurality of images associated with classification labels as candidate training images for a classification model that classifies images to be inspected; a presentation step of presenting at least some of the images among the plurality of images based on an estimation result from the abnormality estimation step; a modification procedure for modifying the classification label of an image for which a modification operation of the classification label has been performed, among the at least some of the images presented by the presentation procedure;

Images