Abnormality detection device and abnormality detection method

Abstract

An abnormality detection device (1) is provided with: an inspection target feature amount extraction unit (121) that generates a first feature amount vector set having, as elements, feature amount vectors of a plurality of partial areas of an inspection target image; an inspection target feature amount conversion unit (122) that converts the first feature amount vector set into a second feature amount vector set that is in the same feature space as a normal feature amount vector set; a normal distance calculation unit (123) that calculates a normal distance for each corresponding element of a set between the normal feature amount vector group set and the second feature amount vector set; an abnormal distance calculation unit (124) that calculates an abnormal distance for each corresponding element of a set between an abnormal feature amount vector group set and the second feature amount vector set; and a determination unit (125) that determines, on the basis of a normal distance set and an abnormal distance set, whether each element of a set corresponding to the inspection target image is normal or abnormal.

Description

Anomaly detection device and anomaly detection method

[0001] The present disclosure relates to an anomaly detection device and an anomaly detection method.

[0002] In recent years, automation of visual inspection of products has progressed. For example, Patent Literature 1 describes an anomaly detection method for automatically detecting an abnormal product from the appearance of an inspection target product. This method uses a normal product learning model that is generated by machine learning using multiple normal image data and is configured in a feature space in which the features of the normal image data are multivariately normally distributed. Identification information for identifying whether a product is normal or abnormal is determined based on the output result when known normal image data and known abnormal image data are input into the normal product learning model. Whether the inspection target product is normal or abnormal is identified based on the identification information for the output result when image data of the inspection target product is input into the normal product learning model.

[0003] Japanese Patent Application Laid-Open No. 2021-174456

[0004] However, conventional anomaly detection methods detect anomalies in the product being inspected based on features extracted from the entire image, which means that it is not possible to identify the location of the anomaly in the image of the product being inspected.

[0005] The present disclosure is intended to solve the above-mentioned problem, and aims to provide an abnormality detection device that can identify the location where an abnormality has occurred in an inspection target image in which an inspection target object has been photographed.

[0006] The anomaly detection device according to the present disclosure includes an inspection object feature extraction unit that extracts a first inspection object feature vector for each of a plurality of partial regions that partition an inspection object image obtained by photographing the inspection object, and generates a first inspection object feature vector set having the first inspection object feature vector as elements; an inspection object feature conversion unit that converts the first inspection object feature vector set into a second inspection object feature vector set that is in the same feature space as the feature vector set for the inspection object in a normal state; a normal feature vector group set that is a collection of a plurality of normal feature vector groups calculated for partial regions at the same position in all normal object images, among the plurality of partial regions that partition a normal object image obtained by photographing the inspection object in a normal state; and a second inspection object feature conversion unit that converts the first inspection object feature vector set into a second inspection object feature vector set that is in the same feature space as the feature vector set for the inspection object in a normal state. The apparatus includes a normal distance calculation unit that calculates a normal distance for each element of a set that corresponds to the second set of inspection object feature vectors and generates a normal distance set whose elements are the normal distances; an abnormal feature vector group set in which a plurality of abnormal feature vector groups calculated for partial regions at the same position in all abnormal object images out of a plurality of partial regions that partition an abnormal object image in which an inspection object in an abnormal state is photographed are collected for all partial regions; and an abnormal distance calculation unit that calculates an abnormal distance for each element of a set that corresponds to the second set of inspection object feature vectors and generates an abnormal distance set whose elements are the abnormal distances; and a determination unit that determines whether each element of the set corresponding to the inspection object image is normal or abnormal based on the normal distance set and the abnormal distance set.

[0007] According to the present disclosure, a normal distance is calculated for each element of a corresponding set between an inspection object feature vector set whose elements are feature vectors of partial regions that divide an inspection object image captured of the inspection object into multiple regions, and a normal feature vector group set whose elements are feature vectors of partial regions that divide a normal object image captured of the inspection object in a normal state into multiple regions. An abnormal distance is calculated for each element of a corresponding set between the inspection object feature vector set and an abnormal feature vector group set whose elements are feature vectors of partial regions that divide an abnormal object image captured of the inspection object in an abnormal state into multiple regions. Whether the inspection object is normal or abnormal is determined for each element of the set corresponding to the inspection object image based on the normal distance and the abnormal distance. Since the elements of the set correspond to partial regions of the image and the presence or absence of an abnormality can be determined for each partial region of the image, the anomaly detection device according to the present disclosure can identify the location of an abnormality in the inspection object image captured of the inspection object.

[0008] 8A is a block diagram showing an example configuration of an anomaly detection device according to embodiment 1. FIG. 8B is a flowchart showing learning processing by the anomaly detection device according to embodiment 1. FIG. 8C is a schematic diagram showing an overview of learning of a neural network classification task using a large-scale image dataset. FIG. 8D is a schematic diagram showing extraction processing of feature vectors by a neural network feature extractor. FIG. 8E is a schematic diagram showing the correspondence between partial regions in a single image and a set of feature vectors. FIG. 8F is a schematic diagram showing the correspondence between partial regions at the same position in a plurality of images and a group of feature vectors. FIG. 8F is a flowchart showing inspection processing by the anomaly detection device according to embodiment 1. FIG. 8A and FIG. 8B are block diagrams showing hardware configurations that realize the functions of the anomaly detection device according to embodiment 1.

[0009] Embodiment 1. Figure 1 is a block diagram showing an example of the configuration of an abnormality detection device 1 according to embodiment 1. In Figure 1, the abnormality detection device 1 acquires an image of an object to be inspected, and automatically determines whether the object to be inspected is normal or abnormal for each of a plurality of partial regions obtained by dividing the image of the object to be inspected. This allows the abnormality detection device 1 to identify the position (partial region) in the image of the object to be inspected where the abnormality has occurred.

[0010] The anomaly detection device 1 uses a feature vector representing the features of each partial region in an image to determine whether an object under inspection is abnormal. For example, the anomaly detection device 1 uses a trained neural network to extract the feature vector from the image. The anomaly detection device 1 is realized using, for example, a tablet terminal, a smartphone, or a laptop personal computer (PC).

[0011] 1 may be connected to the anomaly detection device 1 via a wired or wireless connection. The anomaly detection device 1 is not limited to a device with an external camera device, and may be a device with a built-in camera device. When inspecting the appearance of an object to be inspected, the object to be inspected is photographed by this camera device.

[0012] The anomaly detection device 1 may be a component included in a server capable of communicating with a terminal device. For example, the terminal device may be capable of performing a visual inspection of an inspection object provided in the form of SaaS (Software as a Service). When performing an inspection in the form of SaaS, an inspection application does not need to be installed on the terminal device. The inspection application is executed on the server, and measurement result information is provided to the terminal device on a general-purpose web browser. The inspection application is stored in a storage unit included in the server. The inspection application may also be installed on the terminal device. When the inspection application is installed on the terminal device, the application can be executed to perform a visual inspection of the inspection object.

[0013] The abnormality detection device 1 is realized by a computer including a calculation unit and a storage unit. The storage unit is the storage unit 13 in FIG. 1 , which is a storage device included in the computer. The storage unit 13 includes a storage such as a hard disk drive (HDD) or a solid state drive (SSD), or includes a memory 104 shown in FIG. 8B (described later). Note that the storage unit 13 may be provided external to the abnormality detection device 1 as long as it is accessible by the abnormality detection device 1.

[0014] The calculation unit controls the overall operation of the anomaly detection device 1. The calculation unit includes a learning processing unit 11 and an inspection processing unit 12. The calculation unit executes an inspection application stored in the storage unit 13, thereby realizing various functions of the learning processing unit 11 and the inspection processing unit 12. The learning processing unit 11 includes a normal feature extraction unit 111, a transformation processing matrix generation unit 112, a normal feature conversion unit 113, an abnormal feature extraction unit 114, and an abnormal feature conversion unit 115. The inspection processing unit 12 includes an inspection object feature extraction unit 121, an inspection object feature conversion unit 122, a normal distance calculation unit 123, an abnormal distance calculation unit 124, and a determination unit 125.

[0015] The learning processing unit 11 performs a learning process. In the learning process, the learning processing unit 11 acquires a group of normal object images obtained by photographing an object to be inspected in a normal state and one or more groups of abnormal object images obtained by photographing an object to be inspected in an abnormal state, and generates a set of transformation processing matrices, a second set of normal image feature amount vector groups, and a second set of abnormal image feature amount vector groups.

[0016] The transformation matrix set is a collection of transformation matrices for transforming features in accordance with the distribution of normal object images. The second normal image feature vector group set is a collection of second normal image feature vectors representing features of normal object images after transformation processing. The second abnormal image feature vector group set is a collection of second abnormal image feature vectors representing features of abnormal object images after transformation processing. The transformation matrix set, the second normal image feature vector group set, and the second abnormal image feature vector group set are stored in the memory unit 13 from the learning processing unit 11.

[0017] The inspection processing unit 12 performs an inspection process, which is an appearance inspection of the inspection object. In the inspection process, when the inspection processing unit 12 acquires an inspection object image obtained by photographing the inspection object, the inspection processing unit 12 determines whether the inspection object in the inspection object image is normal or abnormal using the transformation processing matrix set, the second normal image feature amount vector group set, and the second abnormal image feature amount vector group set stored in the storage unit 13.

[0018] First, the learning process according to the first embodiment will be described. Fig. 2 is a flowchart showing the learning process performed by the anomaly detection device according to the first embodiment, illustrating the learning process performed by the learning processing unit 11. The normal feature extraction unit 111 extracts a first set of normal feature vector groups in which multiple normal feature vector groups are collected for all partial regions (step ST1). For example, the normal feature extraction unit 111 trains a neural network using a large-scale image dataset such as ImageNet, and inputs normal object images to a feature extractor of the neural network, thereby extracting the first set of normal feature vector groups from the normal object images.

[0019] FIG. 3 is a schematic diagram showing an overview of learning a classification task of a neural network B using a large-scale image dataset A. The neural network B includes a feature extractor B1 and a classifier B2. In step ST1, the normal feature extraction unit 111 uses the feature extractor B1. The classifier B2 outputs a classification result C. Each element of the first normal feature vector group set corresponds to a partial region obtained by partitioning a normal object image into multiple regions, and a first normal feature vector group corresponds to each partial region. A first normal feature vector, which is an element of the first normal feature vector group, is a feature vector that represents the features of a partial region in one normal object image included in the normal object image group.

[0020] FIG. 4 is a schematic diagram illustrating the extraction process of feature vectors by a neural network feature extractor B1. As shown in FIG. 4, a normal object image A1 is partitioned into multiple partial regions P. The feature extractor B1 may be configured, for example, as a one-layer (L1), two-layer (L2), or three-layer (L3) convolutional neural network (CNN), and a feature vector representing the features of each partial region is extracted in each layer. The first-layer feature vector V1 of the partial region P, the second-layer feature vector V2 of the partial region P, and the third-layer feature vector V3 of the partial region P are combined to form a first normal feature vector VC corresponding to the partial region P. Note that while the feature extractor B1 is illustrated as a three-layer CNN, it is not limited to a three-layer CNN as long as it can extract feature vectors of the partial regions.

[0021] 5 is a schematic diagram showing the correspondence between partial regions P1 to PN in an image A1 and a set of feature vectors. As shown in FIG. 5, image A1 is divided into a plurality of partial regions P1 to PN, and one feature vector is calculated from one of these partial regions. The feature vectors VC1 to VCN of the partial regions P1 to PN are collected together to form the feature vector set VG1.

[0022] 6 is a schematic diagram showing the correspondence between a partial region P1 and a feature vector group VG2 located at the same position in a plurality of images A. As shown in Fig. 6, the image group A includes a plurality of images of the same inspection object. The image group A is, for example, a group of a series of images of inspection target products that are sequentially conveyed on a production line, photographed from the same angle of view.

[0023] The feature extractor B1 extracts feature vectors VC-1 to VC-m from partial regions P1 to Pm that are located at the same position in the image group A. The feature vector group VG2 is a collection of the feature vectors VC-1 to VC-m. In Figures 5 and 6, the feature vector group VG2 is made up of multiple feature vectors VC-1 to VC-m calculated for partial regions that are located at the same positions P1 to Pm in all of the partial regions that divide each image of the image group A into multiple regions, and the collection of these feature vectors for all partial regions P1 to PN is the feature vector group set.

[0024] The transformation matrix generation unit 112 uses the first normal feature vector group set to calculate a transformation matrix in the feature space for each partial region of the normal object image, and generates a transformation matrix set having multiple transformation matrices as elements (step ST2). For example, the transformation matrix generation unit 112 calculates the transformation matrix set by performing singular value decomposition on the first normal feature vector group set for each first normal feature vector group that is an element of the set. The transformation matrix that is an element of the transformation matrix set is calculated for each partial region.

[0025] When the number of elements of the first normal feature vector is P and the number of elements of the first normal feature vector group is N, the singular value decomposition obtains P singular values ​​and a singular value decomposition matrix with P rows and P columns. The transformation matrix generation unit 112 extracts the top K singular values ​​from the P singular values ​​and calculates a P-row, K-column matrix as a transformation matrix by extracting columns corresponding to the top K singular values ​​from the singular value decomposition matrix. Typically, P is a value of about 100 to several hundred. In contrast, K uses a value of about several tens. The transformation matrix set is stored in the storage unit 13.

[0026] The normal feature transform unit 113 transforms the first normal feature vector group set into a normal feature vector group used for calculating the normal distance, which is in the same feature space as the first normal feature vector group set, using the transformation matrix set (step ST3). For example, the normal feature transform unit 113 transforms the first normal feature vector group set for each element of the set using the transformation matrix set, and calculates a second normal feature vector group set. The second normal feature vector group set is stored in the storage unit 13.

[0027] A transformation matrix is ​​calculated for each partial region. A transformation matrix set is a collection of transformation matrices for all partial regions in one image. One normal feature vector is calculated for each partial region in one image. A normal feature vector group has normal feature vectors corresponding to partial regions at the same position in multiple images as its elements. Furthermore, a normal feature vector group set is a collection of normal feature vector groups for all partial regions. The normal feature conversion unit 113 converts the feature space from a number of images × P-dimensional vector to a number of images × K-dimensional vector by multiplying a transformation matrix corresponding to one partial region in the transformation matrix set by the normal feature vector group corresponding to the partial region at the same position.

[0028] The abnormal feature extraction unit 114 extracts a first set of abnormal feature vectors in which a plurality of abnormal feature vectors are collected for all partial regions (step ST4). For example, the abnormal feature extraction unit 114 extracts the first set of abnormal feature vectors by inputting an abnormal object image group to a feature extractor B1 of a trained neural network.

[0029] The abnormal feature quantity converter 115 uses the conversion processing matrix set to convert the first abnormal feature quantity vector group set into an abnormal feature quantity vector group used for calculating the abnormal distance, which is in the same feature space as the first abnormal feature quantity vector group set (step ST5). For example, the abnormal feature quantity converter 115 uses the conversion processing matrix set to convert the first abnormal feature quantity vector group set for each element of the set, and calculates a second abnormal feature quantity vector group set. The second abnormal feature quantity vector group set is stored in the storage unit 13. Note that by also using the conversion processing matrix calculated using the normal feature quantity vector to convert the abnormal feature quantity vector, both the normal feature quantity vector and the abnormal feature quantity vector are converted into the same feature space.

[0030] Next, an anomaly detection method according to the first embodiment will be described. FIG. 7 is a flowchart showing the inspection process by the anomaly detection device 1 according to the first embodiment, and shows the inspection process by the inspection processing unit 12. This inspection process is the anomaly detection method according to the first embodiment. The inspection object feature extraction unit 121 generates a first set of inspection object feature vectors whose elements are feature vectors extracted from the inspection object image (step ST1A). For example, the inspection object feature extraction unit 121 uses the feature extractor B1 to calculate the first set of inspection object feature vectors from the inspection object image in which the inspection object is photographed.

[0031] The inspection-target feature transforming unit 122 transforms the first set of inspection-target feature vectors into a second set of feature vectors (step ST2A). For example, the inspection-target feature transforming unit 122 calculates the second set of inspection-target feature vectors by transforming the first set of inspection-target feature vectors for each element of the set using a transformation processing matrix set read out from the storage unit 13.

[0032] The normal distance calculation unit 123 calculates the normal distance for each element of a corresponding set between the normal feature vector group set and the second test-target feature vector set, and generates a normal distance set whose elements are the normal distances (step ST3A). For example, the normal distance calculation unit 123 calculates the distance between the second normal feature vector group set read from the storage unit 13 and the second test-target feature vector set for each element of the set, thereby calculating the normal distance set. In calculating the normal distance, for example, the mean vector and covariance matrix of the vectors are calculated from the second normal feature vector group, and the Mahalanobis distance between the second test-target feature vector and the second test-target feature vector is calculated.

[0033] The abnormal distance calculation unit 124 calculates the abnormal distance for each element of the corresponding set between the abnormal feature vector group set and the second inspection target feature vector set, thereby generating an abnormal distance set whose elements are the abnormal distances (step ST4A). For example, the abnormal distance calculation unit 124 calculates the distance between the second abnormal feature vector group set read from the storage unit 13 and the second inspection target feature vector set for each element of the set to calculate the abnormal distance set. In calculating the abnormal distance, the inner product of the lower L-dimensional element of the second inspection target feature vector and the lower L-dimensional element of each second abnormal feature vector is calculated, and the value obtained by multiplying this inner product by "-1" is determined as the abnormal distance. Furthermore, the smallest abnormal distance among the abnormal distances between each second abnormal feature vector is determined as the abnormal distance of the corresponding second inspection target feature vector. It has been experimentally confirmed that calculating the inner product by limiting the calculation of the abnormal distance to the "lower L dimensions" in this way increases the correlation between the actual abnormal state and the abnormal distance compared to when no limitation is imposed.

[0034] The determination unit 125 determines whether each element of the set corresponding to the inspection target image is normal or abnormal based on the normal distance set and the abnormal distance set (step ST5A). For example, the determination unit 125 calculates a set of normal and abnormal determination results by performing condition determination on the normal distance set and the abnormal distance set for each element of the set. For example, the determination condition is such that only elements of the set that satisfy both conditions of the normal distance being equal to or less than a first threshold and the abnormal distance being equal to or greater than a second threshold are determined to be normal, and other elements of the set are determined to be abnormal.

[0035] The above-described image training and inspection method provides the following benefits. Because a trained neural network is used to extract features, there is no need to retrain the neural network for each object (Effect 1). The location of an abnormality in an image can be identified to determine whether each subregion in the image is normal or abnormal (Effect 2). By using a transformation matrix that converts a P-dimensional first normal feature into a K-dimensional second normal feature (where P≧K) using singular value decomposition, the time required for visual inspection can be reduced through dimensionality reduction (Effect 3). By using a transformation matrix that converts a P-dimensional first normal feature into a K-dimensional second normal feature (where P≧K) using singular value decomposition, features that accurately represent the characteristics of normal object images can be extracted, improving the accuracy of visual inspection (Effect 4). Because the normal distance is calculated using the K-dimensional second feature that effectively represents the characteristics of normal object images, accurate determination of normality can be performed (Effect 5). In the transformation processing matrix, K dimensions that well represent the features of normal object images are extracted, so for abnormal object images, features that resemble normal object images appear in the upper part of the K dimensions, and features that do not resemble normal object images appear in the lower part. In calculating the abnormal distance, the lower L dimensions of the K dimensions of the second feature vector are used, so that inspection target images that resemble abnormal object images can be searched for with high accuracy (Effect 6).

[0036] Next, we will explain the hardware configuration that realizes the functions of the abnormality detection device 1. The functions of the learning processing unit 11, the inspection processing unit 12, and the storage unit 13 provided in the abnormality detection device 1 are realized by processing circuits. That is, the abnormality detection device 1 has processing circuits for executing the processes of steps ST1A to ST5A shown in Figure 7. The processing circuit may be dedicated hardware, or it may be a CPU (Central Processing Unit) that executes a program stored in memory.

[0037] Fig. 8A is a block diagram showing a hardware configuration that realizes the functions of the abnormality detection device 1. Fig. 8B is a block diagram showing a hardware configuration that executes software that realizes the functions of the abnormality detection device 1. In Figs. 8A and 8B, input interface 100 is an interface that relays image information output from an external device to the abnormality detection device 1. Output interface 101 is an interface that relays abnormality inspection results that are output from the abnormality detection device 1 to the outside.

[0038] 8A , the processing circuit 102 may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a combination thereof. The functions of the learning processing unit 11, the inspection processing unit 12, and the memory unit 13 included in the abnormality detection device 1 may be realized by separate processing circuits, or these functions may be realized together by a single processing circuit.

[0039] When the processing circuit is the processor 103 shown in FIG. 8B , the functions of the learning processing unit 11, the inspection processing unit 12, and the storage unit 13 included in the anomaly detection device 1 are realized by software, firmware, or a combination of software and firmware. The software or firmware is written as a program and stored in the memory 104. The memory 104 is, for example, the storage unit 13 shown in FIG. 1 .

[0040] The processor 103 reads and executes an anomaly detection application program stored in the memory 104, thereby realizing the functions of the learning processing unit 11, the inspection processing unit 12, and the storage unit 13 included in the anomaly detection device 1. For example, the anomaly detection device 1 includes a memory 104 for storing a program that, when executed by the processor 103, results in the processing of steps ST1A to ST5A shown in FIG. 7 being executed. This program causes a computer to execute the procedures or methods of the processing performed by the learning processing unit 11, the inspection processing unit 12, and the storage unit 13. The memory 104 may be a computer-readable storage medium that stores a program for causing a computer to function as the learning processing unit 11, the inspection processing unit 12, and the storage unit 13.

[0041] The memory 104 may be, for example, a non-volatile or volatile semiconductor memory such as a RAM (Random Access Memory), a ROM (Read Only Memory), a flash memory, an EPROM (Erasable Programmable Read Only Memory), or an EEPROM (Electrically-EPROM), a magnetic disk, a flexible disk, an optical disk, a compact disk, a mini disk, or a DVD.

[0042] Some of the functions of the learning processing unit 11, the inspection processing unit 12, and the memory unit 13 included in the anomaly detection device 1 may be realized by dedicated hardware, and the remaining functions may be realized by software or firmware. For example, the memory unit 13 may have its functions realized by a processing circuit 102 that is dedicated hardware, but the learning processing unit 11 and the inspection processing unit 12 may have their functions realized by a processor 103 reading and executing a program stored in a memory 104. In this way, the processing circuit can realize the above functions by hardware, software, firmware, or a combination of these.

[0043] Up to now, the case has been described in which the abnormality detection device 1 includes the learning processing unit 11 and the inspection processing unit 12, but the abnormality detection device 1 may include only the inspection processing unit 12. The learning processing unit 11 and the storage unit 13 may be included in devices provided separately from the abnormality detection device 1. In this case, the abnormality detection device 1 accesses these devices to acquire the learning results.

[0044] As described above, the abnormality detection device 1 according to the first embodiment includes an inspection object feature extraction unit 121 that generates a first inspection object feature vector set whose elements are first inspection object feature vectors extracted from the inspection object image; an inspection object feature conversion unit 122 that converts the first inspection object feature vector set into a second inspection object feature vector set which is in the same feature space as the feature vector set for the inspection object in a normal state; a normal distance calculation unit 123 that calculates a normal distance for each element of a corresponding set between the normal feature vector group set and the second inspection object feature vector set, and generates a normal distance set whose elements are the normal distances; an abnormal distance calculation unit 124 that calculates an abnormal distance for each element of a corresponding set between the abnormal feature vector group set and the second inspection object feature vector set, and generates an abnormal distance set whose elements are the abnormal distances; and a determination unit 125 that determines whether each element of the set corresponding to the inspection object image is normal or abnormal, based on the normal distance set and the abnormal distance set. The elements of the set correspond to partial regions of the image, and the presence or absence of an abnormality can be determined for each partial region of the image, so the abnormality detection device 1 can identify the location of the abnormality in the inspection target image in which the inspection target object is photographed.

[0045] The abnormality detection device 1 according to the first embodiment includes a normal feature extraction unit 111 that extracts a first set of normal feature vectors from a plurality of normal object images; a transformation matrix generation unit 112 that uses the first set of normal feature vectors to calculate a transformation matrix for a feature space for each partial region of the normal object image and generate a transformation matrix set having the plurality of transformation matrices as elements; a normal feature conversion unit 113 that uses the transformation matrix set to convert into a normal feature vector set used for calculating a normal distance, which is in the same feature space as the first set of normal feature vectors; an abnormal feature extraction unit 114 that acquires an abnormal object image group having a plurality of abnormal object images as elements and extracts a first abnormal feature vector set in which, of the plurality of partial regions that partition the abnormal object image, a plurality of abnormal feature vectors calculated for partial regions that are at the same position in all of the abnormal object images are collected for all partial regions; and an abnormal feature conversion unit 115 that uses the transformation matrix set to convert into an abnormal feature vector set used for calculating an abnormal distance, which is in the same feature space as the first set of abnormal feature vectors. This allows the abnormality detection device 1 to learn how to extract feature vectors and calculate transformation processing matrices required for detecting abnormalities in inspection objects.

[0046] In the anomaly detection device 1 according to the first embodiment, the inspection object feature extraction unit 121 is a trained neural network that outputs a first set of inspection object feature vectors when an image is input. By using the trained neural network, the inspection object feature extraction unit 121 can accurately extract the first set of inspection object feature vectors from the image.

[0047] In the anomaly detection device 1 according to the first embodiment, either or both of the normal feature extraction unit 111 and the abnormal feature extraction unit 114 are trained neural networks that, when an image is input, output a set of feature vectors representing the features of each partial region that divides the image into multiple regions. By using the trained neural network, the normal feature extraction unit 111 can accurately extract the set of normal feature vectors from a normal image. By using the trained neural network, the abnormal feature extraction unit 114 can accurately extract the set of abnormal feature vectors from an abnormal image.

[0048] In the anomaly detection device 1 according to the first embodiment, the inspection object feature transform unit 122 uses a transformation matrix set to transform, for each partial region, the first feature vector set into a second feature vector set that is in the same feature space as the feature vector set for the inspection object in a normal state. This enables the inspection object feature transform unit 122 to accurately transform, for each partial region, the first feature vector set into the second feature vector set.

[0049] In the anomaly detection device 1 according to the first embodiment, the transformation matrix generation unit 112 calculates a transformation matrix by performing singular value decomposition on the first normal feature vector group, which is an element of the first normal feature vector group set. By using this transformation matrix, the transformation matrix generation unit 112 can accurately transform the first inspection object feature vector group into a second inspection object feature vector group in the same feature space as the feature vector group for the inspection object in a normal state.

[0050] In the anomaly detection device 1 according to the first embodiment, the normal distance calculation unit 123 calculates the Mahalanobis distance or the Euclidean distance as the normal distance for each element of a corresponding set between the normal feature vector group set and the second feature vector set. This enables the normal distance calculation unit 123 to accurately calculate the normal distance.

[0051] In the anomaly detection device 1 according to the first embodiment, the abnormal distance calculation unit 124 calculates the inner product of the lowest L-dimensional element of the second inspection-target feature vector, which is an element of the second inspection-target feature vector set, and the lowest L-dimensional element of the abnormal feature vector, which is an element of the abnormal feature vector group set, and calculates the inner product value as the abnormal distance by multiplying it by −1. This enables the normal distance calculation unit 123 to accurately calculate the abnormal distance.

[0052] In the abnormality detection device 1 according to the first embodiment, the determination unit 125 determines that the inspection object is in a normal state when the normal distance is equal to or less than the first threshold and the abnormal distance is equal to or greater than the second threshold. This enables the determination unit 125 to determine whether the inspection object is in a normal state for each partial region of the image.

[0053] In the abnormality detection device 1 according to the first embodiment, the determination unit 125 determines that the inspection object is in an abnormal state when the normal distance is equal to or greater than the first threshold and the abnormal distance is equal to or less than the second threshold. This enables the determination unit 125 to determine whether the inspection object is in an abnormal state for each partial region of the image.

[0054] The anomaly detection method according to the first embodiment includes Step ST1A in which an inspection-object feature extraction unit 121 generates a first inspection-object feature vector set, the first inspection-object feature vector set having feature vectors extracted from an inspection-object image as elements, Step ST2A in which an inspection-object feature conversion unit 122 converts the first inspection-object feature vector set into a second feature vector set, Step ST3A in which a normal distance calculation unit 123 calculates a normal distance for each element of a corresponding set between the normal feature vector group set and the second inspection-object feature vector set to generate a normal distance set, Step ST4A in which an abnormal distance calculation unit 124 calculates an abnormal distance for each element of a corresponding set between the abnormal feature vector group set and the second inspection-object feature vector set to generate an abnormal distance set, and Step ST5A in which a determination unit 125 determines whether each element of the set corresponding to the inspection-object image is normal or abnormal based on the normal distance set and the abnormal distance set. By performing this method, the anomaly detection device 1 can identify the location of an abnormality in the inspection-object image.

[0055] The anomaly detection method according to the first embodiment includes step ST1 in which the normal feature extraction unit 111 extracts a first normal feature vector group set in which a plurality of normal feature vector groups are collected for all partial regions; step ST2 in which the transformation matrix generation unit 112 uses the first normal feature vector group set to calculate a transformation matrix for a feature space for each partial region of a normal object image, and generates a transformation matrix set having the plurality of transformation matrices as elements; step ST3 in which the normal feature conversion unit 113 uses the transformation matrix set to convert the first normal feature vector group set into a normal feature vector set used for calculating a normal distance, which is in the same feature space as the first normal feature vector group set; step ST4 in which the abnormal feature extraction unit 114 extracts a first abnormal feature vector group set in which a plurality of abnormal feature vector groups are collected for all partial regions; and step ST5 in which the abnormal feature conversion unit 115 uses the transformation matrix set to convert the first abnormal feature vector group set into an abnormal feature vector set used for calculating an abnormal distance, which is in the same feature space as the first abnormal feature vector group set. By executing this method, the anomaly detection device 1 can learn to extract feature vectors and calculate transformation processing matrices required for detecting an anomaly in an inspection object.

[0056] Any of the components of the embodiments may be modified or omitted.

[0057] The abnormality detection device according to the present disclosure can be used, for example, for visual inspection of products on a production line.

[0058] 1 Anomaly detection device, 11 Learning processing unit, 12 Inspection processing unit, 13 Memory unit, 100 Input interface, 101 Output interface, 102 Processing circuit, 103 Processor, 104 Memory, 111 Normal feature extraction unit, 112 Transformation processing matrix generation unit, 113 Normal feature conversion unit, 114 Abnormal feature extraction unit, 115 Abnormal feature conversion unit, 121 Inspection target feature extraction unit, 122 Inspection target feature conversion unit, 123 Normal distance calculation unit, 124 Abnormal distance calculation unit, 125 Determination unit.

Claims

1. An inspection object feature extraction unit that extracts a first inspection object feature vector from each of a plurality of partial regions that partition an inspection object image obtained by photographing an inspection object, and generates a first inspection object feature vector set having the first inspection object feature vector as elements; an inspection object feature conversion unit that converts the first inspection object feature vector set into a second inspection object feature vector set that is the same feature space as the feature vector set for an inspection object in a normal state; and a normal distance calculation unit that calculates a normal distance for each element of a corresponding set between a normal feature vector group set in which a plurality of normal feature vectors calculated for partial regions at the same position in all normal object images among a plurality of partial regions that partition a normal object image obtained by photographing an inspection object in a normal state, and the second inspection object feature vector set, and generates a normal distance set in which the normal distance is an element. an abnormality detection device comprising: an abnormality distance calculation unit that calculates an abnormality distance for each element of a set that corresponds between an abnormality feature vector group set in which a group of abnormal feature vectors calculated for partial regions at the same position in all of a plurality of partial regions that divide an abnormal object image captured of an object to be inspected in an abnormal state is collected for all of the partial regions, and the second inspection object feature vector set, and generates an abnormality distance set whose elements are the abnormality distances; and a judgment unit that judges whether each element of the set corresponding to the inspection object image is normal or abnormal based on the normal distance set and the abnormal distance set.

2. A normal feature extraction unit that acquires a normal object image group having a plurality of the normal object images as elements, and extracts a first normal feature vector group set in which a plurality of normal feature vector groups calculated for partial regions at the same position in all of the normal object images among a plurality of partial regions that define the normal object image are collected for all partial regions; a transformation processing matrix generation unit that uses the first normal feature vector group set to calculate a transformation processing matrix of a feature space for each partial region of the normal object image, and generates a transformation processing matrix set having the plurality of the transformation processing matrices as elements; a normal feature conversion unit that uses the transformation processing matrix set to convert into a normal feature vector set used for calculating a normal distance, which is the same feature space as the first normal feature vector group set; an abnormal feature extraction unit that acquires an abnormal object image group having a plurality of the abnormal object images as elements, and extracts a first abnormal feature vector group set in which a plurality of abnormal feature vector groups calculated for partial regions at the same position in all of the abnormal object images among a plurality of partial regions that define the abnormal object image are collected for all partial regions; an anomaly detection device according to claim 1, further comprising: an anomaly feature vector conversion unit that converts the first anomaly feature vector group set into an anomaly feature vector set used for calculating an anomaly distance, the anomaly feature vector set being in the same feature space as the first anomaly feature vector group set, by using the conversion processing matrix set.

3. The anomaly detection device according to claim 1, characterized in that the inspection object feature extraction unit is a trained neural network that outputs the first set of inspection object feature vectors when an image is input.

4. The anomaly detection device according to claim 2, characterized in that either or both of the normal feature extraction unit and the abnormal feature extraction unit are trained neural networks that, when an image is input, output a set of feature vectors representing the features of each partial region that divides the image into a plurality of parts.

5. The anomaly detection device according to claim 1, characterized in that the inspection object feature vector transformation unit transforms the first inspection object feature vector set, for each partial region, into a second inspection object feature vector set which is in the same feature space as the feature vector set for the inspection object in a normal state, using a transformation process matrix set.

6. The anomaly detection device according to claim 2, characterized in that the transformation processing matrix generation unit calculates the transformation processing matrix by performing singular value decomposition on the first normal feature vector group, which is an element of the first normal feature vector group set.

7. The anomaly detection device according to any one of claims 1 to 6, characterized in that the normal distance calculation unit calculates a Mahalanobis distance or a Euclidean distance as the normal distance for each element of a corresponding set between the normal feature vector group set and the second inspection target feature vector set.

8. The anomaly detection device according to any one of claims 1 to 7, characterized in that the anomaly distance calculation unit calculates an inner product of an L-dimensional element of the second inspection target feature vector, which is an element of the second inspection target feature vector set, and an L-dimensional element of the anomaly feature vector, which is an element of the anomaly feature vector group set, and multiplies the inner product value by -1 to calculate the anomaly distance.

9. An abnormality detection device as described in any one of claims 1 to 8, characterized in that the judgment unit judges that the inspection object is in a normal state when the normal distance is equal to or less than a first threshold value and the abnormal distance is equal to or greater than a second threshold value.

10. An abnormality detection device as described in any one of claims 1 to 9, characterized in that the judgment unit judges that the inspection object is in an abnormal state when the normal distance is equal to or greater than a first threshold value and the abnormal distance is equal to or less than a second threshold value.

11. An anomaly detection method using an anomaly detection device, comprising: an inspection object feature extraction unit extracting a first inspection object feature vector for each of a plurality of partial regions partitioning an inspection object image obtained by photographing an inspection object, and generating a first inspection object feature vector set having the first inspection object feature vector as elements; an inspection object feature conversion unit converting the first inspection object feature vector set into a second inspection object feature vector set which is the same feature space as the feature vector set for an inspection object in a normal state; and a normal distance calculation unit calculating a normal distance for each element of a corresponding set between a normal feature vector group set in which a plurality of normal feature vectors calculated for partial regions at the same position in all of the normal object images among a plurality of partial regions partitioning a normal object image obtained by photographing an inspection object in a normal state are collected for all partial regions, and the second inspection object feature vector set, and generating a normal distance set in which the normal distance is an element; An abnormality detection method comprising: a step in which an abnormality distance calculation unit calculates an abnormality distance for each element of a set that corresponds between an abnormality feature vector group set in which a group of abnormal feature vectors calculated for partial regions at the same position in all of a plurality of partial regions that divide an abnormal object image in which an inspection object is photographed in an abnormal state is collected for all partial regions, and the second inspection object feature vector set, and generates an abnormality distance set whose elements are the abnormality distances; and a step in which a judgment unit judges whether each element of the set corresponding to the inspection object image is normal or abnormal based on the normal distance set and the abnormal distance set.

12. A normal feature extraction unit obtains a normal object image group having a plurality of the normal object images as elements, and extracts a first normal feature vector group set in which a plurality of normal feature vector groups calculated for partial regions at the same position in all of the normal object images among a plurality of partial regions that divide the normal object image are collected for all partial regions; a transformation processing matrix generation unit uses the first normal feature vector group set to calculate a transformation processing matrix in feature space for each partial region of the normal object image, and generates a transformation processing matrix set having a plurality of the transformation processing matrices as elements; a normal feature conversion unit uses the transformation processing matrix set to convert into a normal feature vector set used for calculating a normal distance, which is the same feature space as the first normal feature vector group set; 12. The anomaly detection method according to claim 11, further comprising: a step of: an abnormality feature extraction unit acquiring an abnormality image group having a plurality of the abnormality images as elements, and extracting a first abnormality feature vector group set in which a plurality of abnormality feature vector groups calculated for partial regions at the same position in all of the abnormality images among a plurality of partial regions dividing the abnormality image are collected for all partial regions; and a step of an abnormality feature conversion unit using the conversion processing matrix set to convert the first abnormality feature vector group set into an abnormality feature vector set used for calculating an abnormality distance, which is in the same feature space as the first abnormality feature vector group set.

Images