Judgment device, inspection system, judgment method, program, and storage medium

Abstract

Provided are a determination device, an inspection system, a determination method, a program, and a storage medium with which the suitability of a classification model can be determined. A determination device according to an embodiment of this invention determines the suitability of a classification model that includes a neural network and outputs a classification result in response to the input of an image. The determination device acquires intermediate data in an intermediate layer of the neural network at the time an input image is input to the classification model. In addition, the determination device uses a plurality of the intermediate data and a plurality of reference data that have been prepared in advance to determine the suitability.

Description

Technical Field

[0001] Embodiments of the present invention relate to a determination device, an inspection system, a determination method, a program, and a storage medium.

Background Art

[0002] A classification model may be used to automatically classify images. For example, the classification model includes a neural network. The accuracy of classification by the classification model can vary depending on the input image.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] The problem to be solved by the present invention is to provide a determination device, an inspection system, a determination method, a program, and a storage medium capable of determining the suitability of a classification model.

Means for Solving the Problems

[0005] The determination device according to the embodiment determines the suitability of a classification model that includes a neural network and outputs a classification result in response to an input of an image. The determination device acquires intermediate data in an intermediate layer of the neural network when an input image is input to the classification model. The determination device further determines the suitability using a plurality of the intermediate data and a plurality of reference data prepared in advance.

Brief Description of the Drawings

[0006] [Figure 1] FIG. 1 is a schematic diagram showing the configuration of an inspection system according to the embodiment. [Figure 2] Figure 2 is a schematic diagram showing the structure of the classification model. [Figure 3] Figure 3 is a flowchart showing the determination method according to the embodiment. [Figure 4] Figure 4 shows the method for determining suitability. [Figure 5] Figure 5 is a schematic diagram illustrating the method for determining suitability. [Figure 6] Figure 6 is a flowchart showing the method for determining suitability in the first modified example. [Figure 7] Figure 7 is a schematic diagram illustrating the method for determining suitability in the first modified example. [Figure 8] Figure 8 is a flowchart showing the method for determining suitability in the second modified example. [Figure 9] Figure 9 is a schematic diagram illustrating an alternative method for determining suitability in a modified example. [Figure 10] Figure 10 is a schematic diagram showing another configuration of the classification model. [Figure 11] Figures 11(a) and 11(b) are schematic diagrams illustrating the effects of distance learning. [Figure 12] Figure 12 is a schematic diagram showing the configuration of another inspection system according to the embodiment. [Figure 13] Figure 13 is a flowchart showing an example of the determination method according to the embodiment. [Figure 14] Figure 14 is a schematic diagram showing the hardware configuration. [Modes for carrying out the invention]

[0007] The embodiments of the present invention will be described below with reference to the drawings. In this specification and in the drawings, elements similar to those already described are denoted by the same reference numerals, and detailed descriptions will be omitted as appropriate.

[0008] Figure 1 is a schematic diagram showing the configuration of the inspection system according to the embodiment. As shown in FIG. 1, the inspection system 1 according to the embodiment includes an imaging device 10, a storage device 20, an inspection device 30, and a determination device 40.

[0009] The imaging device 10 images the article A to be inspected and acquires an image. The imaging device 10 may acquire a video. In this case, a still image is cut out from the video. The imaging device 10 stores the image in the storage device 20.

[0010] The storage device 20 stores a classification model 100 that outputs a classification (class division) result in response to the input of an image. The classification model 100 includes a neural network. The classification model 100 is pre-trained before being used for inspection. The storage device 20 is connected to the imaging device 10, the inspection device 30, and the determination device 40 via a network, wired communication, or wireless communication.

[0011] The inspection device 30 accesses the storage device 20 and acquires the classification model 100 and an input image input to the classification model 100. The inspection device 30 may directly receive the input image from the imaging device 10. The inspection device 30 inspects the article shown in the input image using the classification model 100 and the input image.

[0012] In the inspection, the inspection device 30 inputs the input image to the classification model 100. The inspection device 30 acquires the classification result of the input image from the classification model 100. The classification result corresponds to the inspection result of the article shown in the input image. The inspection device 30 outputs the inspection result of the article according to the classification result.

[0013] By using the image and the classification model 100, the appearance of the article can be accurately inspected without using an expensive inspection device. On the other hand, when the quality of the input image changes, the accuracy of classification by the classification model 100 may also change.

[0014] For example, when the brightness of the space in which the article A and the imaging device 10 are arranged, the relative positional relationship between the article A and the imaging device 10, or the appearance of the article A changes, the quality of the input image changes. Due to the change in the quality of the input image, the inspection accuracy may decrease. That is, the suitability of the classification model 100 for inspection may decrease.

[0015] The determination device 40 determines the suitability of the classification model 100 for inspection. For example, when there is no error in the inspection result but the suitability of the classification model 100 is decreasing due to the change in the quality of the input image, the determination device 40 determines this.

[0016] Hereinafter, the configuration of the classification model 100 and the determination method by the determination device 40 will be described.

[0017] FIG. 2 is a schematic diagram showing the configuration of the classification model. The classification model 100 includes a neural network. This neural network includes, for example, a convolutional neural network (CNN) 110, a fully connected layer 120, and a fully connected layer 130 as shown in FIG. 2.

[0018] An image is input to the CNN 110. The CNN 110 includes a convolutional layer, a pooling layer, etc. The CNN 110 outputs a feature map FM according to the input image. The fully connected layer 120 is provided after the CNN 110. The fully connected layer 120 couples the data of the feature map FM to the nodes of the fully connected layer 130. A plurality of features F are output from the fully connected layer 120. The fully connected layer 130 outputs a result indicating the class of the input image.

[0019] If the classification model 100 can classify images, the learning method of the classification model 100 is arbitrary. For example, the control of the feature space is possible by using distance learning.

[0020] FIG. 3 is a flowchart showing the determination method according to the embodiment. When an input image is input to the classification model 100, not only the classification result but also the data from the intermediate layers of the neural network included in the classification model 100 (intermediate data) is obtained. For example, the output from the fully connected layer 120 is obtained as intermediate data.

[0021] The determination device 40 acquires intermediate data from the classification model 100 (step S10). The determination device 40 acquires intermediate data each time an image is input to the classification model. As a result, multiple intermediate data are acquired. The inspection device 30 may also acquire the intermediate data and store it in the storage device 20. In this case, the determination device 40 acquires the intermediate data from the storage device 20.

[0022] The determination device 40 acquires a set of pre-prepared reference data (step S20). Similar to intermediate data, the reference data is acquired from the intermediate layer when an image is input to the classification model 100. For example, immediately after the classification model has finished training, multiple reference images are sequentially input to the classification model 100. Each reference image is input, and its respective reference data is acquired. The images used for training may also be used as reference data. It is preferable that the images used as reference data have properties equivalent to those used during training and are data acquired immediately before the completion of training.

[0023] The determination device 40 uses the multiple intermediate data obtained in step S10 and the multiple reference data obtained in step S20 to determine the suitability of the classification model 100 (step S30). The determination device 40 outputs the result of the suitability determination of the classification model 100 (step S40).

[0024] The advantages of the embodiment will be explained. As mentioned above, changes in the quality of input images can potentially decrease the accuracy of classification by the classification model. Therefore, it is preferable to update the classification model as needed. For example, one could consider updating the classification model periodically regardless of changes in its accuracy. However, this method carries the risk of unnecessarily updating the classification model and places a heavy burden on updating and managing the model. On the other hand, if the classification model is updated only after accuracy has decreased, inappropriate inspections may be performed until the decrease in accuracy is discovered. For example, many defective products may be distributed, potentially causing damage. Furthermore, inspections cannot be performed between the time the decrease in accuracy is discovered and the time the classification model is updated.

[0025] When the quality of the input image changes, a decrease in the accuracy of the classification model occurs, and a change in the trend of the data output from the intermediate layer of the classification model occurs. This change in trend can occur regardless of the classification result by the classification model, making it easier to detect before the decrease in the accuracy of the classification model becomes significant. By using the data output from the intermediate layer, it is possible to determine at an earlier stage if the suitability of the classification model for inspection has decreased. Even if the accuracy of the classification model decreases, the decrease in suitability can be detected earlier compared to determining the suitability of the classification model from the classification result. For example, by referring to the judgment result from the judgment device 40, the user can more easily determine whether it is appropriate to continue using the classification model 100.

[0026] The determination device 40 according to the embodiment acquires intermediate data in the intermediate layer of the neural network included in the classification model. The determination device 40 then uses the multiple intermediate data and the multiple pre-prepared reference data to determine suitability. This method makes it possible to determine the suitability of the classification model for testing before the accuracy of the classification model deteriorates. By appropriately updating the classification model based on the determination result, there is no need to unnecessarily update the classification model. It becomes possible to update the classification model before its accuracy deteriorates.

[0027] The following sections will describe the specific processing performed by the determination device 40.

[0028] Figure 4 shows the method for determining suitability. In step S30, the determination device 40 calculates a representative value using multiple intermediate data, for example, as shown in Figure 4 (step S31). The representative value is the median or mean of the multiple intermediate data. The representative value may also be a weighted average of the multiple intermediate data.

[0029] The intermediate data used to calculate the representative value is extracted from all the intermediate data. For example, multiple intermediate data obtained within a predetermined period are extracted based on the time the latest intermediate data was obtained. Alternatively, a predetermined number of intermediate data may be extracted in chronological order starting from the latest intermediate data. The predetermined period or predetermined number is set according to changes in intermediate data when the quality of the input image changes, changes in the accuracy of the classification model 100, etc. If there is too much intermediate data used to calculate the representative value, changes in the quality of the input image will not be easily reflected in the representative value. On the other hand, if there is too little intermediate data used to calculate the representative value, the suitability of the classification model 100 may be misjudged due to the influence of errors and outliers.

[0030] The determination device 40 calculates a reference value using multiple reference data (step S32). The reference value is the median or mean of the multiple reference data. The reference value may also be a weighted mean of the multiple reference data. The reference value is calculated using the same calculation method as the representative value. For example, if the representative value is the median of multiple intermediate data, the reference value is the median of the multiple reference data.

[0031] The determination device 40 calculates an evaluation value indicating the similarity between multiple intermediate data and multiple reference data using representative values ​​and reference values ​​(step S33). For example, the evaluation value is the distance between the representative value and the reference value. Euclidean distance or Mahalanobis distance can be used as the distance. The evaluation value may also be an angle between the representative value and the reference value. Cosine similarity can be used as the angle.

[0032] The determination device 40 calculates a comparison value to be compared with the evaluation value using multiple reference data (step S34). The comparison value is set based on the distribution of the multiple reference data. When distance is used as the evaluation value, the comparison value can be set based on the standard deviation, mean squared error, variance, or quartiles (1Q, 3Q) of the distance between the multiple reference data and the reference value. When angle is used as the evaluation value, the comparison value can be set based on the standard deviation, mean squared error, variance, or quartiles (1Q, 3Q) of the angle between the multiple reference data and the origin and the reference value. As an example, the evaluation value is set to three times the standard deviation.

[0033] The determination device 40 compares the evaluation value with the comparison value (step S35). For example, when distance is used as the evaluation value, a smaller evaluation value indicates a smaller change in the trend of the data in the intermediate layer. In this case, the determination device 40 determines whether the evaluation value is smaller than the comparison value. Here, we will explain the case where distance is used as the evaluation value.

[0034] If the evaluation value is smaller than the comparison value, the determination device 40 determines that the suitability of the classification model 100 is in the first state (step S36). For example, this determination result indicates that the quality of the input image has not changed significantly from the reference image, and that the classification model 100 is still suitable for testing.

[0035] If the evaluation value is greater than or equal to the comparison value, the determination device 40 determines that the suitability of the classification model 100 is in the second state (step S37). The second state indicates that the suitability of the classification model 100 is inferior to that of the first state. For example, this determination result indicates that the quality of the input image has changed from the reference image, and there is a high possibility that the accuracy of the classification model 100 will decrease in the near future.

[0036] When cosine similarity is used as the evaluation value for the angle, the closer the evaluation value is to 1 within the range of -1 to 1, the smaller the change in the trend of the data in the intermediate layer. In this case, the determination device 40 determines whether the evaluation value is greater than a comparison value set within the range of -1 to 1. If the evaluation value is greater than the comparison value, the determination device 40 determines that the suitability of the classification model 100 is in the first state. If the evaluation value is less than or equal to the comparison value, the determination device 40 determines that the suitability of the classification model 100 is in the second state.

[0037] In the following sections, we will mainly explain examples where distance is used as an evaluation value, but as mentioned above, angle (cosine similarity) can also be used as an evaluation value. Similarly, when cosine similarity is used as the evaluation value for angle, a corresponding comparison value is set within the range of -1 to 1, and the suitability of classification model 100 is determined based on the relationship between the magnitude of that comparison value and the evaluation value.

[0038] When the determination device 40 determines the suitability of the classification model 100, it outputs the determination result. For example, the determination device 40 sends a notification indicating the determination result to a specific terminal device. The determination device 40 may also save the determination result to the storage device 20. The determination device 40 may send a notification only when a specific determination result is obtained. For example, the determination device 40 sends a notification only when the suitability of the classification model 100 is determined to be in the second state.

[0039] In the above example, the determination device 40 determines the suitability of the classification model 100 to be one of two states. However, the determination device 40 may determine the suitability of the classification model 100 to be one of three or more states, not limited to this example. In this case, multiple comparison values ​​are set. The suitability of the classification model 100 is determined to be one of the states based on the comparison result between the evaluation value and the multiple comparison values.

[0040] In the method described above, the execution order of steps S31 to S34 can be changed as appropriate. If reference values ​​and comparison values ​​have already been calculated during past suitability assessments and these values ​​are available for reference, steps S32 and S34 can be omitted.

[0041] Figure 5 is a schematic diagram illustrating the method for determining suitability. Figure 5 shows multiple intermediate data 210 and multiple reference data 220 in the feature space defined by the first and second features.

[0042] For example, as shown in Figure 5, the determination device 40 calculates a representative value 210a of the multiple intermediate data 210 and a reference value 220a of the multiple reference data 220. The determination device 40 calculates the distance r between the representative value 210a and the reference value 220a. The determination device 40 also calculates three times the standard deviation σ of the multiple reference data 220 as a comparison value. The distance r is greater than the comparison value 3σ. Therefore, the suitability of the classification model 100 is determined to be in the second state.

[0043] (First variation) Alternatively, the determination device 40 may calculate representative values, reference values, etc., according to the image class (classification result). For example, the classification model 100 classifies the input image into either a first class or a second class. The first class indicates that the item in the image is a good product. The second class indicates that the item in the image is a defective product.

[0044] The multiple images that form the basis of the multiple intermediate data include multiple first input images and multiple second input images. The first input images are images that, when input to the classification model 100, are classified into the first class. The second input images are images that, when input to the classification model 100, are classified into the second class.

[0045] Multiple intermediate data sets include multiple first intermediate data sets and multiple second intermediate data sets. Multiple first intermediate data sets are the data output from the intermediate layers when multiple first input images are input to the classification model 100. Multiple second intermediate data sets are the data output from the intermediate layers when multiple second input images are input to the classification model 100.

[0046] Similarly, the multiple reference images that form the basis of multiple reference data include multiple first reference images and multiple second reference images. The first reference image is an image that, when input into the classification model 100, is classified into the first class. The second reference image is an image that, when input into the classification model 100, is classified into the second class.

[0047] Multiple reference data includes multiple first reference data and multiple second reference data. Multiple first reference data are the data output from the hidden layer when multiple first reference images are input to the classification model 100. Multiple second reference data are the data output from the hidden layer when multiple second reference images are input to the classification model 100.

[0048] Figure 6 is a flowchart showing the method for determining suitability in the first modified example. The determination device 40 extracts multiple first intermediate data from multiple intermediate data (step S31a). The determination device 40 calculates a first representative value using the multiple first intermediate data (step S31b). The first representative value is the median or mean of the multiple first intermediate data. The first representative value may also be a weighted mean of the multiple first intermediate data.

[0049] The determination device 40 extracts multiple first reference data from multiple reference data (step S32a). The determination device 40 calculates a first reference value using the multiple first reference data (step S32b). The first reference value is the median or mean of the multiple first reference data. The first reference value may also be a weighted mean of the multiple first reference data. The first reference value is calculated using the same calculation method as the first representative value.

[0050] The determination device 40 calculates a first evaluation value using the first representative value and the first reference value (step S33a). The first evaluation value is the distance or angle between the first representative value and the first reference value.

[0051] The determination device 40 calculates a first comparison value from a plurality of first reference data (step S34a). The first comparison value is set based on the distribution of the plurality of first reference data. For example, the first comparison value is calculated based on the standard deviation, mean squared error, variance, or quartiles (1Q, 3Q) of the plurality of first reference data.

[0052] The determination device 40 determines whether the first evaluation value is smaller than the first comparison value (step S35a). If the first evaluation value is smaller than the first comparison value, the determination device 40 determines that the suitability of the classification model 100 is in the first state (step S36a). If the first evaluation value is greater than or equal to the first comparison value, the determination device 40 determines that the suitability of the classification model 100 is in the second state (step S37a).

[0053] Figure 7 is a schematic diagram illustrating the method for determining suitability in the first modified example. In Figure 7, as in Figure 5, multiple intermediate data 210 and multiple reference data 220 are shown. The multiple intermediate data 210 includes multiple first intermediate data 211 and multiple second intermediate data 212. The multiple reference data 220 includes multiple first reference data 221 and multiple second reference data 222.

[0054] The determination device 40 calculates a first representative value 211a for a plurality of first intermediate data 211 and a first reference value 221a for a plurality of first reference data 221. The determination device 40 calculates the distance r1 between the first representative value 211a and the first reference value 221a. The determination device 40 also calculates three times the standard deviation σ of the plurality of first reference data 221 as the first comparison value. Based on the comparison result between the distance r1 and the first comparison value 3σ, the suitability of the classification model 100 is determined.

[0055] When there is little bias in the number of data points between multiple first intermediate data sets and multiple second intermediate data sets, a method of determining suitability using representative values, reference values, etc., based only on images that fall into one of the classes is effective. Representative values ​​based only on images that fall into one of the classes are more prone to change than representative values ​​based on all the intermediate data. For example, by determining suitability using first evaluation values ​​based only on multiple first intermediate data sets, or first comparison values ​​based only on multiple first reference data sets, it becomes easier to detect a decline in suitability before the accuracy of the classification model deteriorates.

[0056] For example, inspection using classification model 100 is applied to a production line. The items inspected are parts, semi-finished products in the process of being manufactured, or finished products. Typically, many items inspected during manufacturing (especially during mass production) are good products. That is, many input images are classified into a specific class (first class). Therefore, representative values, reference values, comparison values, etc., are greatly influenced by images classified into a specific class. In other words, a representative value based on the entirety of multiple intermediate data can be considered substantially equivalent to a first representative value based on multiple first intermediate data. Similarly, a reference value based on the entirety of multiple reference data can be considered substantially equivalent to a first reference value based on multiple first reference data.

[0057] Representative values ​​and reference values ​​for each class are affected by the accuracy of classification by classification model 100. For example, if the accuracy of classification model 100 decreases, the representative values ​​and reference values ​​for each class will also change. As a result, the detection of a decline in aptitude may be delayed. If the proportion of images classified into a particular class is large, calculating representative values ​​and reference values ​​using the overall intermediate data and overall reference data allows for a more stable determination of a decline in aptitude, regardless of the accuracy of classification model 100.

[0058] (Second variation) In the examples shown in Figures 6 and 7, the suitability of the classification model 100 is determined using only intermediate and reference data from images classified into one class. The suitability of the classification model 100 may also be determined using intermediate and reference data from images classified into other classes. For example, in addition to the process shown in the flowchart of Figure 6 (first determination), the following process (second determination) is further executed.

[0059] Figure 8 is a flowchart showing the method for determining suitability in the second modified example. First, the first determination shown in the flowchart of Figure 6 is performed (step S30a). Next, the determination device 40 extracts multiple second intermediate data from the multiple intermediate data (step S31c). The determination device 40 calculates a second representative value using the multiple second intermediate data (step S31d). The second representative value is the median or mean of the multiple second intermediate data. The second representative value may also be a weighted mean of the multiple second intermediate data.

[0060] The determination device 40 extracts multiple second reference data from multiple reference data (step S32c). The determination device 40 calculates a second reference value using the multiple second reference data (step S32d). The second reference value is the median or mean of the multiple second reference data. The second reference value may also be a weighted mean of the multiple second reference data.

[0061] The second representative value and the second reference value are calculated using the same method as the first representative value and the first reference value.

[0062] The determination device 40 calculates a second evaluation value using the second representative value and the second reference value (step S33b). The second evaluation value is the distance or angle between the second representative value and the second reference value. The second evaluation value is calculated using the same calculation method as the first evaluation value.

[0063] The determination device 40 calculates a second comparison value from a plurality of second reference data (step S34b). The second comparison value is set based on the distribution of the plurality of second reference data. For example, the second comparison value is calculated based on the standard deviation, mean squared error, variance, or quartiles (1Q, 3Q) of the plurality of second reference data. The second comparison value is calculated using the same calculation method as the first comparison value.

[0064] The determination device 40 determines whether the second evaluation value is smaller than the second comparison value (step S35b). If the second evaluation value is smaller than the second comparison value, the determination device 40 determines that the suitability of the classification model 100 is in the first state (step S36b). If the second evaluation value is greater than or equal to the second comparison value, the determination device 40 determines that the suitability of the classification model 100 is in the second state (step S37b).

[0065] The determination device 40 outputs a final suitability determination result based on the suitability determination results of the classification model 100 in the first and second determinations. For example, if the suitability is determined to be in the second state in at least one of the first and second determinations, the determination device 40 determines that the suitability of the classification model 100 is in the second state.

[0066] By determining the suitability of each classification model 100 based on images classified into multiple classes, it is possible to more accurately detect a decline in the suitability of the classification model.

[0067] In the modified examples described above, a value representing the distribution of multiple first intermediate data points may be used as the first evaluation value, and the first comparison value may be set based on the distribution of multiple first reference data points. For example, the standard deviation, mean squared error, variance, or quartiles (1Q, 3Q) of multiple first intermediate data points may be used as the first evaluation value. The first comparison value is set based on the standard deviation, mean squared error, variance, or quartiles (1Q, 3Q) of multiple first reference data points.

[0068] Figure 9 is a schematic diagram illustrating an alternative determination method in the first modified example. In Figure 9, as in Figure 7, multiple intermediate data 210 and multiple reference data 220 are shown. The multiple intermediate data 210 includes multiple first intermediate data 211 and multiple second intermediate data 212. The multiple reference data 220 includes multiple first reference data 221 and multiple second reference data 222. Also in Figure 9, the boundary line BL indicates the boundary between the classification of the first class C1 and the second class C2.

[0069] As the quality of the input image changes, the intermediate data shifts compared to the reference data, as shown in Figure 9. As a result, an image that should be classified as the first class may be classified as the second class. In Figure 9, for example, the intermediate data set 205 contains multiple intermediate data based on images that should be classified as the first class by the classification model 100. However, due to the change in the quality of the input image, the first intermediate data 211 based on images classified as the first class and the second intermediate data 212 based on images classified as the second class are mixed together.

[0070] A shift in intermediate data can result in changes to the distribution of that intermediate data. For example, the distribution of the intermediate data group 205 is represented by a standard deviation of σ0. On the other hand, the distribution of the multiple first intermediate data points 211 is represented by a standard deviation of σ1. Because the intermediate data group 205 straddles the boundary line BL, the distribution of the multiple first intermediate data points 211 becomes smaller than its original distribution.

[0071] The determination device 40 sets comparison values ​​based on the distribution (standard deviation σ2) of the multiple first reference data 221. For example, 0.5 times the standard deviation σ2 and 2 times the standard deviation σ2 are set as comparison values. The determination device 40 calculates the standard deviation σ1 which represents the distribution of the multiple first intermediate data 211. The determination device 40 determines whether the standard deviation σ1 is greater than 0.5 times the standard deviation σ2 and less than 2 times the standard deviation σ2.

[0072] If the standard deviation σ1 is within the range indicated by the comparison value, it indicates that the difference between the distribution of the multiple first intermediate data 211 and the distribution of the multiple first reference data 221 is small. In this case, the determination device 40 determines that the suitability of the classification model 100 is in the first state. If the standard deviation σ1 is outside the range indicated by the comparison value, it indicates that the distribution of the multiple first intermediate data 211 is different from the distribution of the multiple first reference data 221. In this case, the determination device 40 determines that the suitability of the classification model 100 is in the second state.

[0073] Figure 10 is a schematic diagram showing another configuration of the classification model. The determination method described above can be applied to any model from which features can be obtained, not just the classification model 100 shown in Figure 2. For example, the determination method according to the embodiment may be applied to the classification model 100a shown in Figure 10 instead of the classification model 100. The classification model 100a further includes a generative model compared to the classification model 100.

[0074] The feature map FM output from CNN110 is input to the fully connected layer 120 and CNN140, respectively. The function of the fully connected layer 120 in classification model 100a is the same as the function of the fully connected layer 120 in classification model 100.

[0075] The classification model 100a further includes a CNN 140 as a generative model. The CNN 140 reconstructs the feature map FM into data of the same size as the input data and outputs it. In other words, the CNN 110 acts as an encoder and the CNN 140 acts as a decoder.

[0076] The classification device 40 acquires the feature map FM output from the CNN 110 as intermediate data. The classification device 40 uses the acquired intermediate data and pre-prepared reference data to execute the classification method described above.

[0077] For example, CNN110 outputs M feature maps (FM) with an N×N size. The classification device 40 treats the N×N×M dimensional features corresponding to the M feature maps FM as intermediate data and uses them to calculate representative values, evaluation values, etc. Similarly, the classification device 40 treats the N×N×M dimensional features as reference data and uses them to calculate reference values, evaluation values, comparison values, etc. Alternatively, the classification device 40 may calculate representative values ​​for each feature map FM and obtain M dimensional features. The classification device 40 then treats the M dimensional features as intermediate data.

[0078] As shown in Figure 10, the data treated as intermediate data is not limited to the output from the fully connected layer 120, but can also be the output from the CNN 110. Furthermore, the classification model can be modified as needed, as long as it is a model that can obtain features, and is not limited to the examples shown in Figures 2 and 10.

[0079] (study) The classification model described above uses multiple training datasets. For example, the training dataset includes training images and labels indicating the class of those training images. The classification model is trained to output the classification result indicated by the label when a training image is input.

[0080] The classification method according to this embodiment is particularly suitable for distance-learned classification models. In distance learning, the classification model is trained such that in the feature space of the intermediate layer, the distance between data based on images of the same class is short, and the distance between data based on images of different classes is long.

[0081] Figures 11(a) and 11(b) are schematic diagrams illustrating the effects of distance learning. Figure 11(a) shows the first data set 231 and the second data set 232. The first data set 231 is the data of the hidden layer when the first training image is input into the classification model. The first training image is assigned a first label corresponding to the first class. The second data set 232 is the data of the hidden layer when the second training image is input into the classification model. The second training image is assigned a second label corresponding to the second class.

[0082] As a result of distance learning, as shown in Figure 11(b), the distance between two first data points 231 becomes shorter. The distance between two second data points 232 becomes shorter. Also, the distance between the first data point 231 and the second data point 232 becomes longer.

[0083] For specific distance learning methods, Triplet Loss, Contrastive Loss, or ArcFace can be used. As an example, Triplet Loss is used to train a classification model. In Triplet Loss, three images are used as a set for training: a reference Anchor image, a Positive image of the same class as the Anchor, and a Negative image of a different class than the Anchor. The distance dp between Anchor and Positive images and the distance dn between Anchor and Negative images in the feature space are calculated, and the loss function is defined such that dp + α ≤ dn.

[0084] For defective products in mass production, multiple defect modes may exist. Therefore, when using Triplet Loss, it is considered inappropriate to reduce the distance between data within the class indicating defective products (NG class). For this reason, it is preferable to use only images classified as good products (OK class) as anchor images.

[0085] As described above, when distance learning is used, the data distribution in the hidden layers is adjusted in particular. In this embodiment, suitability is determined based on the data trends in the hidden layers. In distance-learned classification models, changes in suitability tend to appear as changes in the data trends in the hidden layers. Therefore, according to this embodiment, changes in the suitability of a distance-learned classification model can be determined with greater accuracy.

[0086] Figure 12 is a schematic diagram showing the configuration of another inspection system according to the embodiment. The inspection system 2 shown in Figure 12 further includes a learning device 50 compared to the inspection system 1 shown in Figure 1. The learning device 50 trains a classification model 100 or 100a. The learning device 50 stores the trained classification model 100 or 100a in the storage device 20.

[0087] As described above, training images, labels, etc., are used for training. Distance learning is preferred. Once training by the training device 50 is complete, testing using the trained classification model 100 or 100a is started. After the start of testing, the judgment device 40 appropriately determines the suitability of the classification model 100 or 100a.

[0088] The judgment device 40 described above can also be applied to classification models used for purposes other than inspection. If a classification model outputs a classification result in response to an image input, the judgment device 40 can determine the suitability of that classification model for classification.

[0089] (Examples) Figure 13 is a flowchart showing an example of the determination method according to the embodiment. In the embodiment, the determination method according to the embodiment is repeatedly executed. As shown in Figure 13, the determination device 40 acquires a plurality of pre-prepared reference data (step S20) and calculates reference values ​​and comparison values ​​(steps S32 and S34). Thereafter, the initially calculated reference values ​​and comparison values ​​are repeatedly used. Each time intermediate data is acquired (step S10), the determination device 40 calculates representative values ​​and evaluation values ​​(steps S31 and S33) and determines the suitability of the model (steps S35 to S37). The determination device 40 also outputs the determination result (step S40).

[0090] As an example, inspection device 30 inspects 100 items per day using classification model 100. Judgment device 40 uses intermediate data obtained over 5 days from the start of inspection using classification model 100 as reference data. Judgment device 40 uses intermediate data from the time the most recent intermediate data was obtained up to 3 days prior. The suitability of classification model 100 is determined using this reference data and intermediate data. Judgment device 40 repeats the suitability determination every day.

[0091] Figure 14 is a schematic diagram showing the hardware configuration. Each of the inspection device 30, the determination device 40, and the learning device 50 includes, for example, the hardware configuration shown in Figure 14. The computer 90 shown in Figure 14 includes a CPU 91, ROM 92, RAM 93, storage device 94, input interface 95, output interface 96, and communication interface 97.

[0092] ROM92 stores programs that control the computer's operation. ROM92 contains the programs necessary for the computer to perform the processes described above. RAM93 functions as a memory area where the programs stored in ROM92 are deployed.

[0093] The CPU 91 includes processing circuits. The CPU 91 uses the RAM 93 as work memory and executes programs stored in at least one of the ROM 92 or the storage device 94. During program execution, the CPU 91 controls each component via the system bus 98 and performs various processes.

[0094] The memory device 94 stores data necessary for program execution and data obtained through program execution.

[0095] The input interface (I / F) 95 connects the computer 90 and the input device 95a. The input I / F 95 is, for example, a serial bus interface such as USB. The CPU 91 can read various data from the input device 95a via the input I / F 95.

[0096] The output interface (I / F) 96 connects the computer 90 to the output device 96a. The output I / F 96 is a video output interface such as a Digital Visual Interface (DVI) or a High-Definition Multimedia Interface (HDMI®). The CPU 91 can send data to the output device 96a via the output I / F 96 and display an image on the output device 96a.

[0097] The communication interface (I / F) 97 connects the computer 90 to a server 97a located outside the computer 90. The communication I / F 97 is, for example, a network card such as a LAN card. The CPU 91 can read various data from the server 97a via the communication I / F 97. The camera 99 takes pictures of objects and saves the images to the server 97a.

[0098] The storage device 94 includes one or more selected from Hard Disk Drives (HDDs) and Solid State Drives (SSDs). The input device 95a includes one or more selected from a mouse, keyboard, microphone (voice input), and touchpad. The output device 96a includes one or more selected from a monitor, projector, speaker, and printer. Devices that have the functions of both input device 95a and output device 96a, such as a touch panel, may also be used.

[0099] The memory device 94 may be used as the memory device 20. The camera 99 may be used as the imaging device 10.

[0100] The functions of the inspection device 30, the determination device 40, and the learning device 50 may be implemented by one or two computers. For example, the inspection device 30 may have the functions of both the determination device 40 and the learning device 50. Alternatively, the functions of the inspection device 30, the determination device 40, and the learning device 50 may be implemented through the collaboration of four or more computers.

[0101] The processing of the various data described above may be recorded as a program that can be executed by a computer on a magnetic disk (flexible disk and hard disk, etc.), an optical disk (CD-ROM, CD-R, CD-RW, DVD-ROM, DVD±R, DVD±RW, etc.), a semiconductor memory, or another non-transitory computer-readable storage medium.

[0102] For example, information recorded on a recording medium can be read by a computer (or embedded system). The recording format (storage format) of the recording medium is arbitrary. For example, a computer reads a program from the recording medium and has the CPU execute the instructions written in the program based on this program. In a computer, program acquisition (or reading) may be performed via a network.

[0103] According to the embodiments described above, a determination device, an inspection system, and a determination method are provided that can determine the suitability of a classification model before its accuracy deteriorates. Furthermore, a similar effect can be achieved by having a computer execute the determination method described above.

[0104] Although several embodiments of the present invention have been illustrated above, these embodiments are presented as examples only and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims of the invention and its equivalents. Furthermore, the embodiments described above can be implemented in combination with each other.

Claims

1. A determination device that includes a neural network and determines the suitability of a classification model that outputs a classification result in response to an image input, When an input image is input to the classification model, intermediate data is obtained in the intermediate layer of the neural network. The system calculates an evaluation value indicating the similarity between the multiple intermediate data and the multiple reference data using a representative value calculated using multiple intermediate data and a reference value calculated using multiple pre-prepared reference data, and determines the suitability by comparing the evaluation value with a comparison value set based on the multiple reference data. Judgment device.

2. The determination device according to claim 1, wherein the classification model is learned by a learning method that allows control over the feature space in the intermediate layer.

3. The classification model is learned by distance learning, as described in claim 1.

4. The plurality of input images include a plurality of first input images that are classified into a first class by the classification model, The plurality of intermediate data includes a plurality of first intermediate data based on the input of the plurality of first input images to the classification model, The determination device according to claim 1, wherein, in determining suitability, a first evaluation value is calculated as the evaluation value using the plurality of first intermediate data.

5. The aforementioned multiple reference data are obtained from the intermediate layer when multiple reference images are input to the classification model, The plurality of reference images include a plurality of first reference images that are classified into the first class by the classification model, The plurality of reference data includes a plurality of first reference data based on the input of the plurality of first reference images to the classification model, The determination device according to claim 4, wherein a first comparison value calculated using the plurality of first reference data is set as the comparison value.

6. The plurality of input images further include a plurality of second input images that are classified into a second class by the classification model, The plurality of intermediate data further includes a plurality of second intermediate data based on the input of the plurality of second input images to the classification model, The plurality of reference images further include a plurality of second reference images that are classified into the second class by the classification model, The plurality of reference data further includes a plurality of second reference data based on the input of each of the plurality of second reference images to the classification model, In the aforementioned suitability determination, further, Using the aforementioned plurality of second intermediate data, a second evaluation value is calculated. The suitability is determined by comparing the second evaluation value with the second comparison value calculated using the plurality of second reference data. The determination device according to claim 5.

7. A determination device according to any one of claims 1 to 6, An imaging device that captures an image of an object and acquires the input image, An inspection device that uses the classification model described above to inspect the items shown in the input image, An inspection system equipped with [the following features].

8. A method for determining the suitability of a classification model that includes a neural network and outputs classification results in response to image input, wherein the computer determines the suitability of the model. To the aforementioned computer, When an input image is input to the classification model, intermediate data is obtained from the intermediate layer of the neural network. An evaluation value indicating the similarity between the multiple intermediate data and the multiple reference data is calculated using a representative value calculated using multiple intermediate data and a reference value calculated using multiple pre-prepared reference data, and the suitability is determined by comparing the evaluation value with a comparison value set based on the multiple reference data. Judgment method.

9. A program that causes the computer to execute the determination method described in claim 8.

10. A storage medium storing the program described in claim 9.

Images