Abnormality inspection device, abnormality inspection method, and abnormality inspection program

Abstract

This abnormality inspection device comprises: an abnormal feature transformation unit (17) that acquires a processing abnormality feature vector based on an abnormal feature vector at a first position on a known abnormal image calculated from the first position; an inspection feature transformation unit (19) that acquires a processing inspection feature vector based on an inspection feature vector of an inspection image to be inspected; an abnormal distance calculation unit (20) that calculates the distance between the processing abnormality feature vector and the processing inspection feature vector at a second position on the inspection image, the second position being different from the first position; and a determination unit (22) that determines whether or not the inspection image is abnormal on the basis of the calculated distance.

Description

Abnormal Inspection Device, Abnormal Inspection Method, and Abnormal Inspection Program

[0001] The present disclosure relates to abnormal inspection technology.

[0002] In recent years, the automation of product appearance inspection has been progressing. For example, Patent Document 1 describes an abnormal determination method for automatically determining abnormal products from the appearance of inspection target products. In this method, a normal product learning model composed of a feature space in which the feature amounts of normal image data are made to follow a multivariate normal distribution, generated by machine learning using a plurality of normal image data, is used. Identification information for identifying whether a product is normal or abnormal is determined based on the output results when known normal image data and known abnormal image data are input to the normal product learning model. Whether the inspection target product is normal or abnormal is identified based on the identification information with respect to the output result when the image data of the inspection target product is input to the normal product learning model.

[0003] Japanese Unexamined Patent Application Publication No. 2021 - 174456

[0004] However, since the conventional abnormal inspection method has identified whether it is normal or abnormal based on the normal product learning model, there has been a problem that abnormal products similar to a specific normal product cannot be identified as abnormal.

[0005] The present disclosure solves the above problems and aims to provide an abnormal inspection technology capable of identifying abnormal products similar to normal products as abnormal.

[0006] One aspect of an abnormal inspection device according to an embodiment of the present disclosure includes an abnormal feature amount conversion unit that acquires a processed abnormal feature amount vector based on an abnormal feature amount vector of a first position calculated from a first position on a known abnormal image, a inspection feature amount conversion unit that acquires a processed inspection feature amount vector based on a inspection feature amount vector of an inspection image to be inspected, an abnormal distance calculation unit that calculates a distance between the processed abnormal feature amount vector and the processed inspection feature amount vector at a second position different from the first position on the inspection image, and a determination unit that determines whether the inspection image is abnormal based on the calculated distance.

[0007] According to the abnormality inspection device of the embodiment of this disclosure, it is possible to identify abnormal products that are similar to normal products as abnormal.

[0008] This is a block diagram showing an example configuration of the abnormality inspection device according to Embodiment 1. This is a diagram showing an example configuration of the hardware of the abnormality inspection device according to Embodiment 1. This is a diagram showing an example configuration of the hardware of the abnormality inspection device according to Embodiment 1. This is a flowchart of the abnormality inspection method according to Embodiment 1.

[0009] Various embodiments of this disclosure will be described in detail below with reference to the drawings. In the drawings, identical or similar parts are denoted by the same or similar reference numerals, and redundant descriptions of such parts will be omitted. In this disclosure, the term "or" is used in the sense of an inclusive OR unless otherwise stated.

[0010] Embodiment 1. <Configuration> An abnormality inspection device according to Embodiment 1 of the present disclosure will be described with reference to Figure 1. Figure 1 is a block diagram showing an example configuration of the abnormality inspection device 1 according to Embodiment 1. In Figure 1, the abnormality inspection device 1 acquires an image of the object to be inspected, and automatically determines whether the object to be inspected is normal or abnormal for each of the multiple sub-regions into which the image to be inspected is divided. As a result, the abnormality inspection device 1 can identify which location (sub-region) of the image to be inspected has an abnormality.

[0011] The anomaly detection device 1 uses feature vectors that represent the characteristics of each sub-region within an image to determine anomalies in the object being inspected. For example, the anomaly detection device 1 uses a pre-trained neural network to extract feature vectors from an image.

[0012] The anomaly detection device 1 can be implemented using, for example, a tablet device, a smartphone, or a notebook-type personal computer (PC).

[0013] An abnormality inspection device 1 may be connected to a camera device (not shown in Figure 1) via a wired or wireless connection. The abnormality inspection device 1 may not be limited to devices with an externally attached camera device; it may also have a built-in camera device. During the visual inspection of the object to be inspected, the object to be inspected is photographed by this camera device.

[0014] The anomaly inspection device 1 may be a component of a server capable of communicating with a terminal device. For example, the terminal device can perform visual inspection of the object to be inspected, provided in the form of SaaS (Software as a Service). When performing inspection in the form of SaaS, the terminal device does not need to have the inspection application installed. The inspection application is executed on the server, and the terminal device is provided with measurement result information on a general-purpose web browser. The inspection application is stored in the storage unit of the server.

[0015] Furthermore, the terminal device may have an inspection application installed. On a terminal device with an inspection application installed, the application can be executed to enable visual inspection of the object being inspected.

[0016] As shown in Figure 1, the anomaly inspection device 1 comprises a feature calculation unit 11, an anomaly feature acquisition unit 12, an anomaly feature storage unit 13, a normal feature acquisition unit 14, a transformation matrix generation unit 15, a transformation matrix storage unit 16, an anomaly feature transformation unit 17, an inspection feature acquisition unit 18, an inspection feature transformation unit 19, an anomaly distance calculation unit 20, a normal distance calculation unit 21, a determination unit 22, and a control unit 10. The control unit 10 controls the overall operation of the anomaly inspection device 1.

[0017] The feature calculation unit 11 calculates and outputs a first abnormal feature vector, a first normal feature vector, or a first inspection feature vector for each sub-region from the abnormal image data, normal image data, or inspection image data. The abnormal image, normal image, or inspection image all contain multiple sub-regions. The feature calculation unit 11 calculates a first abnormal feature vector for each sub-region of the abnormal image from the abnormal image data. The feature calculation unit 11 calculates a first normal feature vector for each sub-region of the normal image from the normal image data. The feature calculation unit 11 calculates a first inspection feature vector for each sub-region of the inspection image from the inspection image data. The calculation of the first abnormal feature vector, the first normal feature vector, and the first inspection feature vector may be performed simultaneously or sequentially. Furthermore, the feature calculation unit 11 may calculate any features for each sub-region. For example, one could train a neural network using a large image dataset such as ImageNet, and then, when image data is input, combine and use the values ​​output from each layer of this neural network.

[0018] The anomaly feature acquisition unit 12 combines the first anomaly feature vector output from the feature calculation unit 11 with the anomaly location (first location) on the anomaly image provided from an external source, and outputs the first anomaly feature vector for the subregion corresponding to the anomaly location on the anomaly image.

[0019] The anomaly feature storage unit 13 is a storage device that stores the first anomaly feature vector output from the anomaly feature acquisition unit 12. This storage device can be an HDD, SSD, or non-volatile memory. It can also be volatile memory as long as the stored contents are retained until the time of inspection. Furthermore, the anomaly feature storage unit 13 does not need to store only one first anomaly feature vector; it may store multiple first anomaly feature vectors.

[0020] The normal feature acquisition unit 14 outputs the first normal feature vector output from the feature calculation unit 11 for each sub-region. Normally, the normal image data of multiple normal images is input to the feature calculation unit 11, and multiple first normal feature vectors are output for each sub-region.

[0021] The transformation matrix generation unit 15 calculates the mean vector for each subregion from multiple first normal feature vectors for each subregion output from the normal feature acquisition unit 14, and calculates the transformation matrix for each subregion using singular value decomposition. The formulas for calculating the mean vector and the transformation matrix are shown in equations (1), (2), and (3).

[0022] μ = avg(X) Equation (1) X: An N x P matrix formed by arranging multiple (N) first normal feature vectors (P-dimensional row (horizontal) vectors) vertically avg(): A calculation that calculates the matrix in column P as the column-direction average of the N x P matrix μ: Average vector (P column vector)

[0023] X - μ = U * Σ * trans(V) Equation (2) -: Operation to subtract the value of the vector in column P from each row of an N x P matrix U: Left singular matrix (N x Q matrix) Σ: Singular value matrix (Q x Q matrix) V: Right singular matrix (P x Q matrix) transpose(): Function to transpose a 2D matrix a * b: Represents the product of matrix a and matrix b N: Number of multiple first normal feature vectors for each subregion P: Dimensions of the first normal feature vector Q: Dimensions of the first normal feature vector after transformation. P ≥ Q. In this disclosure, the singular value matrix is ​​also referred to as the singular value decomposition matrix.

[0024] M = V・Σ Equation (3) M: Conversion matrix (matrix with P rows and Q columns)

[0025] The transformation matrix storage unit 16 is a storage device that stores the mean value vector μ and the transformation matrix M output from the transformation matrix generation unit 15. This storage device can be an HDD, SSD, or non-volatile memory. It can also be volatile memory as long as the stored contents are retained until the time of inspection.

[0026] The anomaly feature transformation unit 17 transforms the plurality of first anomaly feature vectors stored in the anomaly feature storage unit 13 using the mean value vector μ for each subregion and the transformation matrix M stored in the transformation matrix storage unit 16 to generate a second anomaly feature vector. Equation (4) shows the transformation formula. In this disclosure, the second anomaly feature vector may be referred to as a processed anomaly feature vector.

[0027] wA = (xA - μ) * M Equation (4) xA: First anomaly feature vector (P column vector) μ: Mean vector (P column vector) M: Transformation matrix (P row x Q column matrix) wA: Second anomaly feature vector (Q column vector)

[0028] In the transformation, for each subregion, the product of one of the first anomalous feature vectors with the transformation matrix M is calculated to obtain the second anomalous feature vector. If there are K first anomalous feature vectors and Z subregions, a total of K × Z second anomalous feature vectors are calculated.

[0029] The inspection feature acquisition unit 18 outputs the first inspection feature vector output from the feature calculation unit 11 for each sub-region.

[0030] The inspection feature conversion unit 19 converts the first inspection feature vector for each subregion acquired by the inspection feature acquisition unit 18 using the mean value vector μ and the conversion matrix M for each subregion stored in the conversion matrix storage unit 16 to generate a second inspection feature vector. Equation (5) shows the conversion formula. In this disclosure, the second inspection feature vector may be referred to as the processing inspection feature vector.

[0031] w = (x - μ) * M Equation (5) x: First test feature vector (P column vector) μ: Mean vector (P column vector) M: Transformation matrix (P row x Q column matrix) w: Second test feature vector (Q column vector)

[0032] In the transformation, the product of the check feature vectors belonging to the same subregion and the transformation matrix M is calculated to obtain the second check feature vector. Therefore, if the number of subregions is Z, then the number of the second check feature vectors calculated is Z.

[0033] The anomaly distance calculation unit 20 calculates the anomaly distance by performing calculations on the second anomaly feature vector output from the anomaly feature conversion unit 17 and the second check feature vector output from the check feature conversion unit 19. The formula for calculating the anomaly distance can be one in which the distance becomes smaller the more similar the two feature vectors are. For example, it can be the Euclidean distance between the two feature vectors. Alternatively, for example, it can follow formula (6) which uses the dot product between the two feature vectors.

[0034] DA = 1.0 ÷ (MAX(wA・w, ε)) Equation (6) DA: Anomaly distance MAX(): Function to find the maximum value wA: Second anomaly feature vector (Q column vector) w: Second test feature vector (Q column vector) ・: Dot product ε: Very small positive value

[0035] Furthermore, the second anomalous feature vector output from the anomalous feature conversion unit 17 is calculated K times for each of the Z subregions. On the other hand, the second inspection feature vector is calculated one time for each of the Z subregions. Therefore, K anomalous distances are calculated for each of the Z subregions.

[0036] The minimum value of the K abnormal distances may be used as the abnormal distance of the sub-region. Alternatively, the average value of the J smallest abnormal distances among the K abnormal distances may be used as the abnormal distance of the sub-region.

[0037] The normal distance calculation unit 21 calculates the normal distance using the second inspection feature vector w output from the inspection feature conversion unit 19. The formula for calculating the normal distance is shown in equation (7).

[0038] DN = √(w・w) Equation (7) w: Second test feature vector of column Q・: Dot product √(): Square root

[0039] The determination unit 22 combines the abnormal distance for each subregion output from the abnormal distance calculation unit 20 with the normal distance for each subregion output from the normal distance calculation unit 21 to determine whether a subregion is normal or abnormal. For example, an abnormal threshold may be set for the abnormal distance, a normal distance may be set for the normal distance, and the normal / abnormal status may be determined according to the following example of determination conditions.

[0040] Examples of judgment conditions: Normal distance is less than the normal threshold, and abnormal distance is less than the abnormal threshold → Normal Normal distance is less than the normal threshold, and abnormal distance is greater than or equal to the abnormal threshold → Normal Normal distance is greater than or equal to the normal threshold, and abnormal distance is less than the abnormal threshold → Abnormal Normal distance is greater than or equal to the normal threshold, and abnormal distance is greater than or equal to the abnormal threshold → Normal

[0041] As described above, inspecting the inspection image based on the normal and abnormal images yields the following effects. Specifically, even if the abnormal location on the abnormal image (first location) and the inspection location on the inspection image (second location) are different (i.e., even if the normal distance is greater than or equal to the normal threshold), if the content of the abnormality at the abnormal location on the abnormal image and the content of the image at the inspection location on the inspection image are similar (i.e., if the abnormal distance is less than the abnormal threshold), an abnormality can be determined. This allows the abnormality inspection device 1 to identify which location (partial region) of the image under inspection has an abnormality. Since it is possible to determine whether each partial region of the image under inspection is normal or abnormal, even if the object under inspection represented in the image under inspection resembles a normal product, it can be identified as abnormal.

[0042] Next, an example of the hardware configuration of the anomaly inspection device 1 will be described with reference to Figures 2A and 2B. Each function of the anomaly inspection device 1 is realized by a processing circuit. The processing circuit may be a dedicated processing circuit 102 as shown in Figure 2A, or a processor 103 as a computer that executes a program stored in memory 104 as shown in Figure 2B. In Figures 2A and 2B, the input interface 100 is an interface that relays image information, anomaly location information on the anomaly image, and set threshold information output from an external device to the anomaly inspection device 1. The output interface 101 is an interface that relays the anomaly judgment result output from the anomaly inspection device 1 to the outside.

[0043] If the processing circuitry is a dedicated processing circuit 102, the dedicated processing circuit 102 may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (application-specific integrated circuit), an FPGA (field-programmable gate array), or a combination thereof. The functions of the anomaly inspection device 1 may be realized by multiple separate processing circuits, or the functions of the anomaly inspection device 1 may be realized by a single processing circuit.

[0044] When the processing circuitry is the processor 103, the functions of the abnormality inspection device 1 are realized by software, firmware, or a combination of software and firmware. The software and firmware are described as programs and stored in the memory 104. The processor 103 realizes the functions of the abnormality inspection device 1 by reading and executing the programs stored in the memory 104. Here, examples of the memory 104 include non-volatile or volatile semiconductor memories such as RAM (random access memory), ROM (read-only memory), flash memory, EPROM (erasable programmable read only memory), EEPROM (electrically erasable programmable read-only memory), etc., as well as magnetic disks, flexible disks, optical disks, compact disks, mini-disks, and DVDs.

[0045] Note that part of the functions of the abnormality inspection device 1 may be realized by dedicated hardware, and the other functions may be realized by software or firmware. Thus, the processing circuitry can realize the functions of the abnormality inspection device 1 by hardware, software, firmware, or a combination thereof.

[0046] <Operation> Next, the operation of the abnormality inspection device 1 will be described with reference to FIG. 3.

[0047] (Step ST1) In step ST1, the abnormality feature quantity conversion unit 17 acquires a processed abnormality feature quantity vector based on the abnormality feature quantity vector at the first position calculated from the first position on the known abnormal image. Specifically, the abnormality feature quantity conversion unit 17 converts the first abnormal feature quantity vector acquired by the abnormality feature quantity acquisition unit 12 using the average value vector μ and the conversion matrix M, and acquires the converted vector as the processed abnormality feature quantity vector.

[0048] (Step ST2) In step ST2, the inspection feature quantity conversion unit 19 acquires a processed inspection feature quantity vector based on the inspection feature quantity vector of the inspection image for inspection. Specifically, the inspection feature quantity conversion unit 19 converts the first inspection feature quantity vector acquired by the inspection feature quantity acquisition unit 18 with the average value vector μ and the conversion matrix M, and acquires the converted vector as the processed inspection feature quantity vector (second inspection feature quantity vector). Note that for the processing of step ST1 and step ST2, either one of the processes may be performed first, or both processes may be performed simultaneously.

[0049] (Step ST3) In step ST3, the abnormal distance calculation unit 20 calculates the distance between the processed abnormal feature quantity vector and the processed inspection feature quantity vector at a second position different from the first position on the inspection image.

[0050] (Step ST4) In step ST4, the determination unit 22 determines whether the inspection image is abnormal based on the distance calculated by the abnormal distance calculation unit 20.

[0051] Note that it is possible to combine the embodiments, or appropriately modify or omit each embodiment.

[0052] The abnormal inspection device according to the present disclosure can be used, for example, for the appearance inspection of products on a production line.

[0053] 1 Abnormal inspection device, 10 Control unit, 11 Feature quantity calculation unit, 12 Abnormal feature quantity acquisition unit, 13 Abnormal feature quantity storage unit, 14 Normal feature quantity acquisition unit, 15 Conversion matrix generation unit, 16 Conversion matrix storage unit, 17 Abnormal feature quantity conversion unit, 18 Inspection feature quantity acquisition unit, 19 Inspection feature quantity conversion unit, 20 Abnormal distance calculation unit, 21 Normal distance calculation unit, 22 Determination unit.

Claims

1. An anomaly inspection device comprising: an anomaly feature transformation unit that acquires a processed anomaly feature vector based on an anomaly feature vector of a first position calculated from a first position on a known anomaly image; an inspection feature transformation unit that acquires a processed inspection feature vector based on an inspection feature vector of an inspection image to be inspected; an anomaly distance calculation unit that calculates the distance between the processed anomaly feature vector and a processed inspection feature vector at a second position on the inspection image that is different from the first position; and a determination unit that determines whether the inspection image is anomaly based on the calculated distance.

2. The abnormality inspection apparatus according to claim 1, wherein the inspection feature conversion unit calculates the processed inspection feature vector by singular value decomposition of the inspection feature vector using a singular value decomposition matrix calculated from normal feature vectors obtained from a plurality of normal images, and the abnormality feature conversion unit calculates the processed abnormality feature vector by singular value decomposition of the abnormal feature vector using the singular value decomposition matrix.

3. An anomaly detection device according to claim 2, further comprising: a normal feature acquisition unit that acquires the normal feature vector from the plurality of normal images; and a transformation matrix generation unit that calculates the singular value decomposition matrix from the normal feature vector.

4. The abnormality inspection device according to claim 3, further comprising a normal distance calculation unit that calculates the distance between the processing inspection feature vector and the processing inspection quantity vector from the normal distribution represented by the singular value decomposition matrix by referring to the processing inspection feature vector and the singular value decomposition matrix, wherein the determination unit determines an abnormality in the image data based on the distance calculated by the normal distance calculation unit.

5. The abnormality inspection apparatus according to any one of claims 1 to 4, wherein the abnormality feature conversion unit calculates a processed inspection feature vector using a singular value decomposition matrix calculated from normal feature vectors obtained from a plurality of normal images.

6. The anomaly inspection device according to any one of claims 1 to 5, wherein the inspection feature conversion unit calculates the processed inspection feature vector by decomposing the inspection feature vector using the singular value decomposition matrix.

7. An anomaly inspection device according to any one of claims 1 to 6, wherein the anomaly feature vector is a first anomaly feature vector, and the processed anomaly feature vector is a second anomaly feature vector calculated from a second position on the anomaly image that is different from the first position.

8. An anomaly inspection method performed by an anomaly inspection device comprising an anomaly feature conversion unit, an inspection feature conversion unit, an anomaly distance calculation unit, and a determination unit, comprising: a step in which the anomaly feature conversion unit obtains a processed anomaly feature vector based on an anomaly feature vector of a first position calculated from a first position on a known anomaly image; a step in which the inspection feature conversion unit obtains a processed inspection feature vector based on an inspection feature vector of an inspection image to be inspected; a step in which the anomaly distance calculation unit calculates the distance between the processed anomaly feature vector and a processed inspection feature vector at a second position on the inspection image that is different from the first position; and a step in which the determination unit determines whether the inspection image is anomaly based on the calculated distance.

9. An anomaly inspection program that causes a computer to perform the following functions: a function to acquire a processed anomaly feature vector based on an anomaly feature vector of a first position calculated from a first position on a known anomaly image; a function to acquire a processed inspection feature vector based on an inspection feature vector of an inspection image to be inspected; a function to calculate the distance between the processed anomaly feature vector and a processed inspection feature vector at a second position on the inspection image that is different from the first position; and a function to determine whether the inspection image is anomaly based on the calculated distance.

Images