Method, system, and program for detecting abnormal regions

Abstract

To provide a method of detecting an abnormal area in which excessive detection is reduced.SOLUTION: A method of detecting an abnormal area included in an object image by using a feature extraction model configured as a convolution neural network having a plurality of feature extraction layers, includes the steps of: (a) preparing a plurality of non-defective product feature quantities; (b) extracting an object feature quantity for each of a plurality of patches; (c) acquiring k representative points which are representative points in each of k clusters by clustering the plurality of non-defective product feature quantities; and (d) creating an abnormality degree map by calculating, as an abnormality degree, the nearest neighbor distance between the object feature quantity and the k representative points about each of the plurality of patches. In the step (c), clustering is performed multiple times so that the object may be different, and the k representative points are acquired having wider distribution than the k representative points acquired when performing the clustering once so as to acquire the k representative points.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to a method, a system, and a program for detecting an abnormal region. [Background technology]

[0002] Non-Patent Document 1 discloses a technology for detecting anomalies by acquiring a set of feature values ​​of good products and feature values ​​of input data for inspection on a patch-by-patch basis from a model trained on ImageNet, and measuring the distance between the two in the feature space. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Niv Cohen, Yedid Hoshen “Sub-Image Anomaly Detection with Deep Pyramid Correspondences”, arXiv:2005.02357, Wed, 3 Feb 2021. Summary of the Invention [Problem to be solved by the invention]

[0004] The images of non-defective products used to acquire the feature sets contain a biased background area outside the inspection target and specific non-defective product patterns. If the features resulting from such areas are acquired and used for anomaly detection, there is a high possibility of overdetection of non-defective samples. [Means for solving the problem]

[0005] According to one aspect of the present disclosure, there is provided a method for detecting an abnormal region. This method is a method for detecting an abnormal region included in a target image using a feature extraction model configured as a convolutional neural network having multiple feature extraction layers, and includes the steps of: (a) preparing multiple good-product features obtained from each of the multiple feature extraction layers when multiple good-product images are input into the feature extraction model; (b) inputting the target image into the feature extraction model and, in each of the multiple feature extraction layers, extracting target features, which are features of the target image, for each of multiple patches corresponding to multiple patch images obtained by dividing the target image; (c) clustering the multiple good-product features to obtain k representative points, which are representative points of each of k clusters; and (d) creating a stratified abnormality map by calculating, for each feature extraction layer of the multiple feature extraction layers, the nearest distance between the target feature and the k representative points for each of the multiple patches as the degree of abnormality, wherein in the step (c), clustering is performed multiple times for different targets to obtain k representative points that are more widely distributed than the k representative points obtained by performing clustering once. [Brief explanation of the drawings]

[0006] [Figure 1] FIG. 1 is a block diagram showing the configuration of an anomaly detection system. [Figure 2] FIG. 2 is an explanatory diagram showing an example of the configuration of a feature extraction model. [Figure 3] FIG. 10 is a diagram showing a flow of creating a stratified abnormality map. [Figure 4] 10 is a flowchart showing the procedure of an abnormality detection process. [Figure 5] 10 is a flowchart showing the procedure of a non-defective product feature extraction process. [Figure 6] An equation explaining the non-defective product feature extraction process. [Figure 7] FIG. 10 is a diagram illustrating a representative point acquisition process. [Figure 8] FIG. 10 is a diagram showing a case where a representative value is obtained in a comparative example. [Figure 9]10 is a flowchart showing the procedure of a representative point acquisition process. [Figure 10] An equation that explains the representative point acquisition process. [Figure 11] 6 is a histogram showing the distribution of abnormality degrees in the first example and the first comparative example. [Figure 12] 10 is a histogram showing the distribution of anomaly levels in the second embodiment. [Figure 13] 10 is a histogram showing the abnormality degree distribution of the second comparative example. [Figure 14] 10 is a flowchart showing the procedure of an abnormality detection process according to the second embodiment. [Figure 15] 10 is a flowchart showing the procedure of a representative point acquisition process according to the second embodiment. [Figure 16] An example of the GUI displayed on the display. DETAILED DESCRIPTION OF THE INVENTION

[0007] A. First embodiment: A1. System Configuration: Fig. 1 is a block diagram showing the configuration of an anomaly detection system that executes the anomaly detection process shown in Fig. 4. As shown in Fig. 1, the anomaly detection system includes an information processing device 100 and a camera 400. The camera 400 captures an image of an object for anomaly detection. The information processing device 100 functions as an anomaly detection device that detects anomalies contained in the image captured by the camera 400. The information processing device 100 can be realized, for example, by a personal computer.

[0008] The information processing device 100 has a processor 110, a memory 120, an interface circuit 130, and an input device 140 and a display unit 150 connected to the interface circuit 130. For example, but not limited to, the processor 110 not only has the function of executing the processes described in detail below, but also has the function of displaying data obtained by the processes and data generated in the process of the processes on the display unit 150. A camera 400 is also connected to the interface circuit 130.

[0009] The processor 110 functions as an anomaly detection unit 112 that detects anomalies contained in an image. The function of the anomaly detection unit 112 is realized by the processor 110 executing a computer program stored in the memory 120. However, the function of the anomaly detection unit 112 may also be realized by a dedicated electronic circuit. The term "processor" used in this specification includes such a dedicated electronic circuit. Furthermore, the processor that executes the processing of the anomaly detection unit 112 may be a processor included in a remote computer connected to the information processing device 100 via a network. Furthermore, the processing of the anomaly detection unit 112 may be executed by multiple processors.

[0010] The memory 120 stores a feature extraction model 200, a good product image group GM, and a good product feature group GF. The feature extraction model 200 is a machine learning model that finds features from images and has been pre-trained with an image database. Specifically, ResNet, which has been pre-trained with ImageNet, can be used as the feature extraction model 200. The good product image group GM is a group of images of good products that do not contain defective areas. The good product feature group GF is composed of features obtained when each good product image in the good product image group GM is input into the feature extraction model 200.

[0011] FIG. 2 is an explanatory diagram showing an example configuration of a feature extraction model 200. As shown in FIG. 2, the feature extraction model 200 is a convolutional neural network in which an input layer that receives an input image IM is followed by a first feature extraction layer FE1, a second feature extraction layer FE2, a third feature extraction layer FE3, and an average pooling layer 240, arranged in this order. The first feature extraction layer FE1, the second feature extraction layer FE2, and the third feature extraction layer FE3 are collectively referred to as feature extraction layers FE. Of the four layers FE1 to FE3, 240, the first feature extraction layer FE1 is the lowest layer, and the average pooling layer 240 is the highest layer. Although the example in FIG. 2 uses three feature extraction layers FE, the number of feature extraction layers FE is arbitrary. However, it is preferable to use two or more feature extraction layers FE.

[0012] The three feature extraction layers FE are convolutional layers that each extract features of the input image IM. In the example of Figure 2, the plane size of the input image IM is 224 x 224 pixels. The first feature extraction layer FE1 has a plane size of 56 x 56 nodes and a channel count of Z1. Nodes refer to neurons. The second feature extraction layer FE2 has a plane size of 28 x 28 and a channel count of Z2. The third feature extraction layer FE3 has a plane size of 14 x 14 and a channel count of Z3. The channel counts Z1, Z2, and Z3 of each layer can be set to any number greater than or equal to 1. A 16 x 16 pixel image in the input image IM is called a patch image PT0 or patch PT0. Patch PT0 in the input image IM corresponds to a 4 x 4 patch PT1 in the first feature extraction layer FE1, a 2 x 2 patch PT2 in the second feature extraction layer FE2, and a 1 x 1 patch PT3 in the third feature extraction layer FE3.

[0013] In the anomaly detection process, features are extracted for each patch PT1 from the outputs of multiple nodes belonging to the first feature extraction layer FE1. The features for one patch PT1 in the first feature extraction layer FE1 are represented by the outputs of nodes with a size equal to [plane size of patch PT1] × [number of channels], i.e., 4 × 4 × Z1 nodes. Therefore, for example, the feature corresponding to one patch PT1 is a vector consisting of 4 × 4 × Z1 elements. Similarly, the second feature extraction layer FE2 extracts features for each patch PT2, and the third feature extraction layer FE3 extracts features for each patch PT3. Therefore, each of the three feature extraction layers FE extracts features from 14 × 14 patches. Because the feature extraction layer FE corresponds to an intermediate layer, the features extracted from the feature extraction layer FE can also be called "intermediate features."

[0014] The average pooling layer 240 performs a calculation to obtain a 1×1×Z3 output by calculating the average value of the outputs from the 14×14 nodes of the third feature extraction layer FE3 for each channel. The output of the average pooling layer 240 is a feature vector representing the features of the entire image. However, since the output of the average pooling layer 240 is not used in the anomaly detection processing according to this embodiment, it is possible to omit the average pooling layer 240.

[0015] A2. Overview of Anomaly Detection Process: In the anomaly detection process, a first stratified anomaly degree map AM1, a second stratified anomaly degree map AM2, and a third stratified anomaly degree map AM3 are created, each corresponding to one of the three feature extraction layers FE. The first stratified anomaly degree map AM1, the second stratified anomaly degree map AM2, and the third stratified anomaly degree map AM3 are collectively referred to as the stratified anomaly degree maps AM.

[0016] Fig. 3 is a diagram showing the flow of creating one stratified anomaly degree map AM. As described above, the good-quality image group GM shown in Fig. 3 is a group of images of good products. The group of feature amounts F obtained when each good-quality image in the good-quality image group GM is input to the feature extraction model 200 is the good-quality feature amount group GF.

[0017] Similar to a non-defective product image, a target image TM to be inspected is input to the feature extraction model 200, and a feature F is extracted for each patch. The extracted feature F is called a target feature TF.

[0018] The stratified anomaly map AM is roughly created by calculating the degree of similarity between the target feature TF and the feature F of the non-defective feature group GF for each patch and arranging the calculated degrees of similarity at the patch positions. Specifically, the degree of similarity here refers to, for example, cosine similarity or Pearson similarity, or distance such as Euclidean distance or Manhattan distance. The higher the degree of similarity, the higher the degree of similarity. The shorter the distance, the higher the degree of similarity. In this specification, the term "distance" is used to mean the degree of similarity. In other words, in this specification, distance is a concept that includes similarity.

[0019] In this embodiment, to determine the degree of similarity between a target feature TF and a feature F of the good-quality feature group GF, the nearest neighbor distance, which is the distance between the target feature TF and the feature F of the good-quality feature group GF that is most similar to the target feature TF, is used. Because the number of data items in the good-quality feature group GF is enormous, calculating the distance between all feature items F constituting the good-quality feature group GF and the target feature TF to determine the nearest neighbor distance would require a huge amount of computation. Therefore, in this embodiment, the distance between the representative point RP obtained by clustering and the target feature TF is calculated. This reduces the computational load required to calculate the degree of similarity. Note that this method of calculating the nearest neighbor distance between the representative point RP of the good-quality feature group GF and the target feature TF, instead of the nearest neighbor distance between the feature F of the good-quality feature group GF and the target feature TF, is also called approximate nearest neighbor search.

[0020] More specifically, the nearest neighbor distance between the representative point RP of each group obtained by grouping the non-defective feature group GF through clustering and each of the multiple target features TF is calculated on a patch-by-patch basis. The average of the calculated multiple nearest neighbor distances for each patch is arranged at the patch position to create a stratified anomaly map AM, which is a heat map. This makes it possible to visualize positions determined to be abnormal. The stratified anomaly map AM is color-coded according to the length of the nearest neighbor distance. The larger the nearest neighbor distance, the lower the degree of similarity and the higher the degree of abnormality.

[0021] The images in the non-defective product image group GM shown in FIG. 3 are images of the adhesive surface AS of an electronic component package. The adhesive surface AS is the target inspection area. As shown in FIG. 3, the image used for inspection also includes a background area outside the inspection area. A feature F is also extracted from the background area. If the background area occupies a large proportion of the image, multiple feature F extracted from the background area are likely to remain in multiple groups even after clustering. Therefore, the feature F extracted from the background area is included in the distance calculation even though it is a part unnecessary for inspection. Therefore, in this embodiment, a method for acquiring the representative point RP has been devised. This makes it possible to prevent feature F that does not contribute to inspection from being included in the distance calculation.

[0022] The inventors also noticed that the frequency of occurrence of feature quantities F in the non-defective product feature quantity group GF is not uniform, and that the degree of abnormality calculated using feature quantities F with low occurrence frequencies tends to be high. Here, low occurrence frequency means that the number of feature quantities F that are relatively similar to each other in the distribution of feature quantities F is small. Therefore, in this embodiment, a method for acquiring representative points RP using clustering is devised. This makes it possible to suppress overdetection, in which an abnormality is detected despite the fact that the product is normal. Note that, among the feature quantities F in the non-defective product feature quantity group GF, feature quantities F with low occurrence frequencies include feature quantities F extracted from images with indefinite patterns, such as stains.

[0023] A3. Anomaly detection process: FIG. 4 is a flowchart showing the procedure for anomaly detection processing. FIG. 5 is a flowchart showing the procedure for good-product feature extraction processing. FIG. 6 is a formula explaining the good-product feature extraction processing. FIG. 9 is a flowchart showing the procedure for representative point acquisition processing. FIG. 10 is a formula explaining the representative point acquisition processing. In step S10 shown in FIG. 4, the anomaly detection unit 112 inputs a plurality of good-product image groups GM to the feature extraction model 200, extracts feature amounts F, and prepares a good-product feature group GF.

[0024] 5, in the good product feature extraction process, a feature F is extracted for each feature extraction layer FE for one good product image in the good product image group GM. Specifically, step S101, which is the process of extracting a feature F for one good product image, is repeated until extraction is completed for all feature extraction layers FE. Then, extraction of the feature F for each feature extraction layer FE is repeated until extraction of the feature F for all good product images included in the good product image group GM is completed.

[0025] The good product feature extraction process is shown by formula (A) in Fig. 6. Step S101 in Fig. 5 is shown by formula (a1) in Fig. 6. In formula (a1), for one good product image in the good product image group GM, the feature F extracted on a patch-by-patch basis for each feature extraction layer FE is converted into data without spatial information and added to the good product feature group GF of each feature extraction layer FE.

[0026] In step S20 of FIG. 4, similarly to step S10, the anomaly detection unit 112 inputs the target image to the feature extraction model 200 and extracts the target feature TF.

[0027] In step S30, a representative point acquisition process is performed to acquire representative points RP of the non-defective feature group GF. FIG. 7 is a diagram illustrating the representative point acquisition process in step S30. In step S30, clustering is performed multiple times for different objects until k representative points RP are acquired. The number of clusters and representative points acquired in one clustering is k. sub There are individuals.

[0028] As shown in Figure 7, the first clustering is performed on the good product feature set GF, and k sub After the first clustering, k representative points RP are obtained. subThe feature F for which at least one of the distances to each of the representative points RP is equal to or less than the threshold th is excluded from the next clustering. This allows the next clustering to be performed with redundant feature F, which are highly similar to each other and appear frequently, removed. Then, in the second clustering, feature F for which all distances to each of the representative points RP are greater than the threshold th, is selected as the target. sub representative points RP are acquired. Similarly, for the third and subsequent clustering operations, clustering is repeated while thinning out the feature amount F until the total number of acquired representative points RP reaches k. In this way, by performing clustering multiple times while thinning out the feature amount F, it is possible to acquire a variety of representative points RP that are widely distributed and have low similarity to one another.

[0029] FIG. 8 shows a comparative example in which clustering is performed once and k sub This is a diagram showing the case where 2 × representative values ​​are obtained. When clustering is performed once, representative points RP are obtained that reflect the feature values ​​F with high appearance frequencies. Therefore, representative points RP with a higher degree of similarity to each other and a smaller spread are obtained compared to when clustering is performed multiple times while thinning out the feature values ​​F.

[0030] As described above, when the image used for inspection has a large proportion of background regions and the feature F extracted from the background regions has a high frequency of appearance, by applying the representative point acquisition method according to this embodiment, it is possible to reduce the influence of the feature F that appears frequently and acquire the representative point RP.

[0031] Specifically, in the representative point acquisition process, clustering is performed for each feature extraction layer FE for all feature extraction layers FE, as shown in FIG. 9. In this embodiment, the clustering is performed using the k-means method. In step S311, k sub In step S312, the k representative points RP are clustered. subOf the distances to each of the representative points RP, the feature F whose closest distance is greater than a predetermined threshold th is set as the target of the next clustering. This makes it possible to exclude the feature F near the representative points RP obtained by clustering from the targets of the next clustering. Then, clustering is repeated until the total number of representative points RP reaches k. Then, when acquisition of k representative points RP has been completed for all feature extraction layers, this subroutine ends. In this embodiment, the center of gravity of the feature F within the cluster is used as the representative point RP.

[0032] The representative point acquisition process is represented by equation (B) in FIG. 10. Step S311 in FIG. 9 is represented by equation (b1) in FIG. 10. Step S312 in FIG. 9 is represented by equations (b2) and (b3) in FIG. 10. In equation (b1), ksub representative points RP are acquired from the non-defective product feature group GF. In equation (b2), the newly acquired representative point RP is added to the group of representative points RP. In equation (b3), the feature F whose minimum distance from each of the multiple representative points RP is greater than a threshold th is set as the target of the next clustering.

[0033] As described above, similarity can be used as the degree of similarity. Similarity is an index that approaches 1 as the degree of similarity increases. When similarity is used as the degree of similarity, it is possible to convert similarity into distance using equation (1) and apply equation (b3) in FIG. 10. In this embodiment, an index obtained by subtracting cosine similarity from 1, as shown in equation (1), is used as the distance in equation (b3).

number

[0034] In step S40 of FIG. 4, the anomaly detection unit 112 calculates the nearest neighbor distance between the representative point RP and each of the plurality of target features TF for each patch, and then calculates the average of the calculated nearest neighbor distances. The nearest neighbor distance is calculated by a method in which distances are calculated in a round-robin fashion and the minimum value is taken as the nearest neighbor distance. In this embodiment, as in step S312, the index obtained by subtracting the cosine similarity from 1 in equation (1) is used as the distance. As described above, the acquired representative point RP reflects not only feature values ​​F with high occurrence frequencies but also feature values ​​F with low occurrence frequencies. Therefore, by using the nearest neighbor distance between the representative point RP and the target feature value TF as the degree of anomaly, the frequency with which the degree of anomaly is calculated as high for feature values ​​F with low occurrence frequencies is reduced, thereby suppressing overdetection.

[0035] In step S50, the anomaly detection unit 112 arranges the nearest neighbor distances at patch positions to create a stratified anomaly degree map AM as an anomaly degree map. As a result, as shown in the lower part of FIG. 2, three stratified anomaly degree maps AM are created for each of the three feature extraction layers FE. The first stratified anomaly degree map AM1 is a map that shows the anomaly degree for each patch PT1. Similarly, the second stratified anomaly degree map AM2 is a map that shows the anomaly degree for each patch PT2, and the third stratified anomaly degree map AM3 is a map that shows the anomaly degree for each patch PT3.

[0036] In step S100, the abnormality detection unit 112 displays the stratified abnormality degree map AM on the display unit 150, and then this processing routine ends. By displaying the stratified abnormality degree map AM on the display unit 150, the user can check abnormal areas with a high degree of abnormality.

[0037] A4. Experimental result 1: FIG. 11 is a histogram showing the anomaly distribution for non-defective images of the first example and the first comparative example. The anomaly distribution shown in FIG. 11 is a graph in which representative points RP are obtained using the non-defective image group GM, and the maximum values ​​of the anomaly map for each of the non-defective images constituting the non-defective image group GM are tallied. The horizontal axis of the anomaly distribution represents the degree of anomaly, and the vertical axis represents the frequency. The first example represents the result when the anomaly detection process according to this embodiment is applied, clustering is performed four times with thinning, and 2000 representative points RP are obtained. The first comparative example represents the result when clustering is performed once, instead of step S30 according to this embodiment, and 2000 representative points RP are obtained.

[0038] 11, the distribution of the degree of abnormality in Example 1 is shifted to the low degree of abnormality side compared to Comparative Example 1. This shows that by using the method for acquiring representative points RP according to this embodiment, it is possible to acquire representative points RP that reflect the feature values ​​F that appear less frequently for non-defective images.

[0039] A5. Experimental result 2: FIG. 12 is a histogram showing the distribution of the degree of anomaly between the good and defective images of the second example. FIG. 13 is a histogram showing the distribution of the degree of anomaly between the good and defective images of the second comparative example. The second example and the second comparative example are graphs showing the degree of anomaly calculated for each patch of the good and defective images by acquiring a representative point RP using the good image group GM. The horizontal axis of the anomaly distribution represents the degree of anomaly, and the vertical axis represents frequency. Comparing FIG. 12 with FIG. 13, it can be seen that the distribution of the second example is shifted toward a lower degree of anomaly compared to the second comparative example. This is because, as in Experimental Result 1, by using the method for acquiring the representative point RP according to this embodiment, it was possible to acquire a representative point RP for the good images that reflects the feature value F, which appears less frequently.

[0040] The second example and the second comparative example are results obtained using the same good product images and defective product images. The number of good product images used as training data to extract the good product feature group GF was 735. The number of defective product images used as test data to calculate the degree of abnormality was 2,397, and the number of defective product images was 6. In the second example, the number of representative points RP per run was 500, and the total number of representative points RP was 2,000. In the second comparative example, the total number of representative points RP was 2,000. The distance calculation in step S312 in the second example used an index obtained by subtracting the cosine similarity from 1. The threshold value th in step S312 in the second example was 0.1.

[0041] Table 1 shows the results of the AUROC, false positive rate, and number of false positives for the second example and the second comparative example. The false positive rate is the percentage of images judged to be defective among images of good products. As shown in Table 1, the second example has a higher AUROC and a lower false positive rate compared to the second comparative example. According to the method of this embodiment, it is possible to obtain a variety of representative points RP from the feature values ​​F of good products, which shows that it is possible to accurately distinguish between good and defective products. The "0% oversight" shown in Figures 12 and 13 indicates the position of the smallest degree of abnormality among the degrees of abnormality obtained for defective product images. The "0% oversight" can be used as a guideline when determining a threshold for determining whether an inspected product is defective. [Table 1]

[0042] According to the first embodiment described above, in step S30, clustering is performed multiple times on the non-defective product feature group GF to obtain k representative points RP for different targets, and k representative points RP that are more widely distributed than the k representative points RP obtained by performing clustering once are obtained. This makes it possible to detect abnormal areas using a variety of representative points RP. Therefore, the frequency with which feature values ​​F resulting from background areas are used for abnormality detection decreases, allowing for accurate abnormality detection.

[0043] Furthermore, in step S30, the second and subsequent clustering is performed on the feature values ​​F whose distance from the representative point RP is greater than the threshold value th for each representative point RP obtained by the previous clustering. Therefore, the feature values ​​F that are highly similar to each other can be excluded from clustering, and a variety of representative points RP can be obtained.

[0044] B. Second embodiment: FIG. 14 is a flowchart showing the procedure of an anomaly detection process according to this embodiment. FIG. 15 is a flowchart showing the procedure of a representative point acquisition process according to this embodiment. FIG. 16 is an example of a GUI (Graphical User Interface) displayed on the display unit 150. The anomaly detection process according to this embodiment differs from the anomaly detection process according to the first embodiment in the method of determining the threshold value th, in creating an integrated anomaly degree map AMT, and in accepting changes to the threshold value th used to thin out the feature amount F in clustering. Steps that are the same as the processing steps according to the first embodiment are given the same reference numerals, and detailed explanations will be omitted as appropriate.

[0045] 14, similarly to the first embodiment, steps S10 to S30 are performed by the anomaly detection unit 112. Step S30 according to this embodiment differs from the first embodiment in that it includes a processing step of determining a threshold value th used for thinning out the feature amount F, as shown in FIG.

[0046] 15, the anomaly detection unit 112 determines the threshold value th using the distribution of the feature values ​​F of the non-defective feature value group GF. Specifically, there are a method of using the average value of a patch distance group, which is a set of distances between two feature values ​​F for each patch, a method of using the standard deviation of the patch distance group, a method of using the distance that falls in the top 10% in descending order of distance in the distribution of the average values ​​of the patch distance group, and a method of appropriately combining these methods. By determining the threshold value th using the distribution, the distribution status of the feature values ​​F can be reflected in the threshold value th. In step S312, the threshold value th determined in step S301 is used.

[0047] As shown in FIG. 14 , steps S40 to S50 are performed by the anomaly detection unit 112, as in the first embodiment. In step S60, the anomaly detection unit 112 performs weighted addition of the stratified anomaly degree maps AM1, AM2, and AM3 to create an integrated anomaly degree map AMT, as shown in FIG. 16 . The integrated anomaly degree map AMT is created by first adjusting the resolution of the stratified anomaly degree maps AM1, AM2, and AM3 to make them equal, and then adding the stratified anomaly degree maps AM1, AM2, and AM3 after the resolution adjustment. When adding the stratified anomaly degree maps AM1, AM2, and AM3, the weights of the stratified anomaly degree maps AM1, AM2, and AM3 may be added equally or unequal. Here, unequal means that the weights are different from each other. The resolution adjustment is preferably performed to match the size of the input image IM to the feature extraction model 200. In the example of Figure 2, the size of the input image IM is 224 x 224 pixels, so the size of the stratified abnormality maps AM1, AM2, and AM3 after resolution adjustment will also be 224 x 224 pixels. When adjusting the resolution, pixel values ​​may be interpolated. Blur processing may also be performed using a blurring filter such as a Gaussian filter before or after adding together the stratified abnormality maps AM1, AM2, and AM3.

[0048] 12, the anomaly detection unit 112 detects, as an abnormality location, a patch in the integrated anomaly degree map AMT whose degree of abnormality is equal to or greater than a predetermined anomaly degree threshold. In step S110, the anomaly detection unit 112 identifies the detected abnormality location and causes the stratified anomaly degree map AM and the integrated anomaly degree map AMT to be displayed on the display unit 150. Specifically, the anomaly detection unit 112 displays the abnormality location in a color different from that of other locations, for example, red, in the stratified anomaly degree map AM and the integrated anomaly degree map AMT. This allows the user to confirm the abnormality location.

[0049] As shown in FIG. 16, a window 500 displays a target image TM, an integrated abnormality degree map AMT, a first strata-specific abnormality degree map AM1, a second strata-specific abnormality degree map AM2, and a third strata-specific abnormality degree map AM3. The window 500 also displays a slider 501 and a recalculation button 502. The slider 501 accepts changes to the threshold value th. The recalculation button 502 is initially displayed as grayed out and unselectable. When the threshold value th is changed from its initial state, the recalculation button 502 is displayed in an enabled, selectable state. When the user wants to change the threshold value th, he or she moves the slider 501 and then selects the recalculation button 502, which is now enabled.

[0050] 14, the anomaly detection unit 112 determines whether a change in the threshold value th has been accepted. Specifically, if the recalculation button 502 shown in Fig. 16 has been selected, it is determined that a change in the threshold value th has been accepted. On the other hand, if the recalculation button 502 is displayed in an unselectable state and has not been selected, it is determined that a change in the threshold value th has been accepted.

[0051] In step S120, if the anomaly detection unit 112 determines that a change to the threshold value th has not been received, it terminates this processing routine. On the other hand, if the anomaly detection unit 112 determines that a change to the threshold value th has been received, it performs the processing from step S30 to step S110 using the threshold value th received using the slider 501. This allows the user to adjust the slider 501 while viewing the displayed stratified anomaly degree map AM and integrated anomaly degree map AMT. For example, if the target image TM contains a stain that is not a defect, the degree of anomaly at the scratched area may be displayed as high. In this case, the user can change the threshold value th, check the displayed degree of anomaly, and then change the threshold value th.

[0052] According to the second embodiment described above, in step S60, the integrated abnormality degree map AMT is created, and in step S110, the abnormal locations are displayed on the display unit 150. This allows the user to visually confirm the abnormal locations.

[0053] Furthermore, in step S301, the threshold value th is determined using the distribution of the target feature value TF. This allows the threshold value th to be a value that reflects the distribution of the target feature value TF. Furthermore, if the threshold value th is changed in step S120, the anomaly detection unit 112 performs steps S30 to S110 again. This allows the user to adjust the threshold value th by looking at the stratified anomaly degree map AM and the integrated anomaly degree map AMT.

[0054] C. Other Embodiments: (C1) Other embodiment 1: In the first embodiment, the anomaly detection process ends after step S100 is executed. However, as in the second embodiment, after step S50 is executed, the process of creating an integrated anomaly degree map AMT, the process of determining whether or not there is an anomaly, the process of accepting a change to the threshold value th, and the process steps from step S30 onwards may be executed again using the accepted threshold value th.

[0055] (C2) Alternative embodiment 2: In the first embodiment, the stratified abnormality degree map AM is used by being displayed on the display unit 150. The method of using the stratified abnormality degree map AM is not limited to displaying it on the display unit 150 and using it as an image. For example, the target image TM can be classified using the positions of patches having an abnormality degree equal to or greater than the abnormality threshold, and used for analyzing defects. Furthermore, the stratified abnormality degree map AM may be used to determine that a target image TM in which the positions of patches having an abnormality degree equal to or greater than the abnormality threshold are at specific positions is defective.

[0056] (C3) Alternative embodiment 3: In the first embodiment, the k-means method is used for clustering, but other methods, such as the single link method, may also be used.

[0057] (C4) Alternative embodiment 4: In the first embodiment, the number of representative points extracted in one clustering is the same for all clusterings. sub The number of representative points extracted in one clustering run may not be the same for multiple clustering runs, but may be different from each other. Regardless of the number of representative points, a variety of representative points RP can be obtained by performing clustering while thinning out the feature amount F.

[0058] (C5) Alternative embodiment 5: In the first embodiment, in step S30, the non-defective feature values ​​to be clustered are thinned out, thereby performing clustering so that the clustering targets are different. The method of varying the clustering targets is not limited to thinning out. The clustering targets may be varied by dividing the space of the non-defective feature value group GF in advance and performing clustering for each divided space. In this case as well, a variety of representative points RP can be obtained.

[0059] D. Other forms: The present disclosure is not limited to the above-described embodiments and can be realized in various configurations without departing from the spirit thereof. For example, the technical features of the embodiments corresponding to the technical features in each aspect described below can be appropriately replaced or combined to solve some or all of the above-described problems or achieve some or all of the above-described effects. Furthermore, if a technical feature is not described as essential in this specification, it can be appropriately deleted.

[0060] (1) According to a first aspect of the present disclosure, there is provided a method for detecting an abnormal region. This method is a method for detecting an abnormal region included in a target image using a feature extraction model configured as a convolutional neural network having multiple feature extraction layers, and includes the steps of: (a) preparing multiple good-product features obtained from each of the multiple feature extraction layers when multiple good-product images are input into the feature extraction model; (b) inputting the target image into the feature extraction model and, in each of the multiple feature extraction layers, extracting target features, which are features of the target image, for each of multiple patches corresponding to multiple patch images obtained by dividing the target image; (c) clustering the multiple good-product features to obtain k representative points, which are representative points of each of k clusters; and (d) creating an anomaly degree map by calculating, for each feature extraction layer of the multiple feature extraction layers, the nearest distance between the target feature and the k representative points for each of the multiple patches as the degree of anomaly, wherein in the step (c), clustering is performed multiple times for different targets to obtain k representative points that are more widely distributed than the k representative points obtained by performing clustering once. According to this embodiment, since a variety of representative points are acquired, the frequency with which features resulting from background areas other than the inspection area included in the image of the inspection target are used for abnormality detection is reduced, enabling accurate abnormality detection.

[0061] (2) In the above embodiment, in step (c), the second and subsequent clustering may be performed on the non-defective feature values ​​whose degree of similarity to the representative points obtained by the previous clustering is less than a threshold value for each of the representative points obtained by the previous clustering. According to this embodiment, clustering can be performed excluding feature values ​​similar to the representative points obtained by the previous clustering. Therefore, a variety of representative points can be obtained, and anomaly detection can be performed with high accuracy.

[0062] (3) In the above embodiment, after step (d), the method may further include the steps of: (e) adjusting the resolution of the stratified anomaly degree maps obtained for the feature extraction layers so that the resolutions thereof are equal to each other, and adding the stratified anomaly degree maps after the resolution adjustment to create an integrated anomaly degree map; (f) detecting, as an abnormality location, a patch in the integrated anomaly degree map whose anomaly degree is equal to or greater than an anomaly degree threshold; and (g) displaying the abnormality location on a display unit. According to this embodiment, a user can check an abnormality area by looking at the integrated anomaly degree map displayed on the display unit.

[0063] (4) In the above embodiment, the threshold value may be predetermined, and after step (g), a step (h) of accepting a change to the threshold value may be performed, and steps (c) through (g) may be performed again using the accepted threshold value. According to this embodiment, the use of a predetermined threshold value makes it easier to display abnormalities. The user can freely change the threshold value and check the abnormality area after the threshold value change by looking at the abnormality area displayed on the display unit.

[0064] (5) In the above embodiment, before step (c), a step (x) of determining the threshold value using the distribution of the non-defective feature quantities may be further included, where the threshold value is the value determined in step (x). After step (g), a step (h) of accepting a change to the threshold value may be further included, and steps (c) through (g) may be performed again using the accepted threshold value. According to this embodiment, the threshold value can be determined by reflecting the distribution of non-defective feature quantities. The user can then freely change the threshold value and check the abnormality location after the threshold value change by looking at the abnormality region displayed on the display unit.

[0065] The present disclosure can be realized in various forms other than a method for detecting an abnormal region, such as a system for detecting an abnormal region, a computer program for realizing a method for detecting an abnormal region, or a non-transitory recording medium on which the computer program is recorded. [Explanation of symbols]

[0066] AM1...first stratified anomaly map, AM2...second stratified anomaly map, AM3...third stratified anomaly map, AMT...integrated anomaly map, F...feature, FE1...first feature extraction layer, FE2...second feature extraction layer, FE3...third feature extraction layer, GF...group of good product features, GM...group of good product images, IM...input image, PT0, PT1, PT2, PT3...patches, 100...information processing device, 110...processor, 112...anomaly detection unit, 120...memory, 130...interface circuit, 140...input device, 150...display unit, 200...feature extraction model, 240...average pooling layer, 400...camera, 500...window, 501...slider, 502...recalculation button, AS...adhesion surface, RP...representative point, TF...target feature

Claims

1. A method for detecting an abnormal region included in a target image using a feature extraction model configured as a convolutional neural network having multiple feature extraction layers, comprising: (a) preparing a plurality of non-defective product features obtained from each of the plurality of feature extraction layers when a plurality of non-defective product images are input to the feature extraction model; (b) inputting the target image into the feature extraction model, and extracting, in each of the plurality of feature extraction layers, target features that are features of the target image for each of a plurality of patches corresponding to a plurality of patch images obtained by dividing the target image; (c) clustering the plurality of non-defective feature quantities to obtain k representative points that are representative points of the k clusters; (d) creating an anomaly degree map by calculating, for each of the plurality of patches in each of the plurality of feature extraction layers, the nearest neighbor distance between the target feature and the k representative points as an anomaly degree; In step (c), clustering is performed multiple times so that the objects are different, and k representative points are obtained that are more widely distributed than the k representative points obtained when clustering is performed once.

2. 10. The method of claim 1, In the step (c), the second and subsequent clustering operations are performed on the non-defective feature values ​​whose degree of similarity to each representative point obtained by the previous clustering operation is smaller than a threshold value.

3. 3. The method of claim 2, further comprising, after step (d), (e) performing resolution adjustment to equalize the resolutions of the plurality of anomaly degree maps obtained for the plurality of feature extraction layers, and creating an integrated anomaly degree map by adding the plurality of anomaly degree maps after the resolution adjustment; (f) detecting, in the integrated anomaly degree map, patches whose anomaly degrees are equal to or greater than an anomaly degree threshold as an abnormal portion; (g) displaying the abnormality on a display unit.

4. 4. The method of claim 3, The threshold value is predetermined, and after the step (g), (h) accepting a change to the threshold value; and performing steps (c) to (g) again using the received threshold value.

5. 4. The method of claim 3, further comprising, before step (c), (x) determining the threshold value using a distribution of the non-defective feature quantities; The threshold value is a value determined in the step (x), and after the step (g), (h) accepting a change to the threshold value; and performing steps (c) to (g) again using the received threshold value.

6. A system for detecting an abnormal region included in a target image using a feature extraction model configured as a convolutional neural network having multiple feature extraction layers, a memory that stores the feature extraction model and a plurality of non-defective product features obtained from each of the plurality of feature extraction layers when a plurality of non-defective product images are input to the feature extraction model; one or more processors for performing operations using the feature extraction model; Equipped with The processor: (a) preparing a plurality of non-defective product features obtained from each of the plurality of feature extraction layers when a plurality of non-defective product images are input to the feature extraction model; (b) inputting the target image into the feature extraction model, and extracting, in each of the plurality of feature extraction layers, target features that are features of the target image for each of a plurality of patches corresponding to a plurality of patch images obtained by dividing the target image; (c) clustering the plurality of non-defective feature quantities to obtain k representative points that are representative points of the k clusters; (d) creating an anomaly degree map by calculating, for each of the plurality of patches, the nearest neighbor distance between the target feature and the k representative points as an anomaly degree for each of the plurality of patches in the plurality of feature extraction layers; In step (c), the system performs clustering multiple times so that the objects are different, and obtains k representative points that are more widely distributed than the k representative points obtained when clustering is performed once to obtain k representative points.

7. A program for detecting an abnormal region included in a target image using a feature extraction model configured as a convolutional neural network having a plurality of feature extraction layers, (a) a process of preparing a plurality of non-defective product features obtained from each of the plurality of feature extraction layers when a plurality of non-defective product images are input to the feature extraction model; (b) inputting the target image into the feature extraction model, and extracting, in each of the plurality of feature extraction layers, target feature amounts that are feature amounts of the target image for each of a plurality of patches corresponding to a plurality of patch images obtained by dividing the target image; (c) clustering the plurality of non-defective feature quantities to acquire k representative points that are representative points of the k clusters; (d) creating an anomaly degree map by calculating, for each of the plurality of patches in each of the plurality of feature extraction layers, a nearest neighbor distance between the target feature and the k representative points as an anomaly degree; In the process (c), the program performs clustering multiple times for different objects to obtain k representative points that are more widely distributed than the representative points obtained when clustering is performed once.

Images