Image inspection device, image inspection method, and image inspection program

Abstract

This image inspection device comprises: a training data set acquisition unit for acquiring a training data set; a training feature amount derivation unit for deriving training feature amounts of training image data; a clustering processing unit for clustering the training image data on the basis of the training feature amounts and dividing the same into a plurality of clusters; an index derivation unit for deriving, for each cluster, the center of gravity and distance of a plurality of the training feature amounts belonging to the cluster, and deriving an average mu and a variance sigma of a plurality of the distances corresponding to the plurality of training feature amounts; an imaging feature amount derivation unit for extracting imaging feature amounts on the basis of imaging data; a minimum distance derivation unit for deriving each distance between the imaging feature amounts and the plurality of centers of gravity and deriving a minimum distance X that is the minimum among the distances; an anomaly score calculation unit for calculating an anomaly score alpha on the basis of the minimum distance X, the average mu, and the variance sigma; and a comparison unit for comparing the anomaly score alpha with a preset threshold value.

Description

Image inspection device, image inspection method, and image inspection program

[0001] The present invention relates to an image inspection device, an image inspection method, and an image inspection program.

[0002] Conventionally, in image inspection of products, in order to prevent defective products from being mistakenly judged as good products, there has been an image inspection method that combines multiple judgment algorithms to make a final pass / fail judgment on the product (for example, Patent Document 1).

[0003] However, with conventional image inspection methods, the judgment scores can be unstable and it is not possible to clearly distinguish between good and bad products using a specific threshold. Therefore, there is room for further improvement in the accuracy of the final pass / fail judgment of products.

[0004] Japanese Patent No. 6999150

[0005] An object of the present invention is to provide an image inspection device, an image inspection method, and an image inspection program that have high accuracy in pass / fail judgment.

[0006] (1) An image inspection device according to one aspect of the present invention includes a training dataset acquisition unit that acquires a training dataset including a plurality of training image data; a training feature derivation unit that derives training features of the training image data; a clustering processing unit that clusters the training image data based on the training features to divide the training image data into a plurality of clusters; an index derivation unit that derives, for each cluster, a center of gravity of the plurality of training features belonging to the cluster and a distance between the center of gravity and the training feature, and derives an average μ and a variance σ of the plurality of distances corresponding to the plurality of training features; an imaging feature derivation unit that extracts imaging features based on imaging data captured by a camera; a minimum distance derivation unit that derives each distance between the imaging feature and the plurality of centers of gravity and derives a minimum distance X that is the smallest of the distances; an abnormality degree calculation unit that calculates an abnormality degree α based on the minimum distance X, the average μ, and the variance σ; and a comparison unit that compares the abnormality degree α with a predetermined threshold. (2) In (1) above, the device may further include an error processing unit that executes error processing when the degree of abnormality α exceeds the threshold. (3) In (1) or (2) above, the clustering processing unit may perform the clustering multiple times using random numbers, and the index derivation unit may derive the center of gravity for each cluster each time the clustering is performed, and may use a representative cluster that minimizes the residual sum of squares of the distances in the cluster as a population for calculating the minimum distance X. (4) In (1) or (2) above, the feature derivation unit may perform resizing processing, window function processing, FFT processing, or band-pass filter processing.(5) An image inspection method according to one aspect of the present invention includes: a training dataset acquisition step of acquiring a training dataset including a plurality of training image data; a training feature derivation step of deriving training features of the training image data; a clustering processing step of clustering the training image data based on the training features to divide the training image data into a plurality of clusters; an index derivation step of deriving, for each cluster, a center of gravity of the plurality of training features belonging to the cluster and a distance between the center of gravity and the training feature, and deriving an average μ and a variance σ of the plurality of distances corresponding to the plurality of training features; an imaging feature derivation step of extracting imaging features based on imaging data captured by a camera; a minimum distance derivation step of deriving each distance between the imaging feature and the plurality of centers of gravity and deriving a minimum distance X that is the smallest of the distances; an abnormality degree calculation step of calculating an abnormality degree α based on the minimum distance X, the average μ, and the variance σ; and a comparison step of comparing the abnormality degree α with a predetermined threshold. (6) An image inspection program according to one aspect of the present invention causes a computer to execute: a training dataset acquisition function that acquires a training dataset including multiple pieces of training image data; a training feature derivation function that derives training features of the training image data; a clustering processing function that clusters the training image data based on the training features to divide the training image data into multiple clusters; an index derivation function that derives, for each cluster, the center of gravity of the multiple training features belonging to the cluster and the distance between the center of gravity and the training features, and derives the average μ and variance σ of the multiple distances corresponding to the multiple training features; an imaging feature derivation function that extracts imaging features based on imaging data captured by a camera; a minimum distance derivation function that derives each distance between the imaging features and the multiple centers of gravity and derives a minimum distance X that is the smallest of the distances; an abnormality degree calculation function that calculates an abnormality degree α based on the minimum distance X, the average μ, and the variance σ; and a comparison function that compares the abnormality degree α with a preset threshold.

[0007] According to the present invention, it is possible to provide an image inspection device, an image inspection method, and an image inspection program that have high accuracy in determining whether an object is pass or fail.

[0008] It is a diagram for explaining an outline of an image inspection device. It is a diagram for explaining an outline of an image inspection method. It is a diagram for explaining a flow of an image inspection method. It is a diagram for explaining a flow of a feature amount derivation step.

[0009] (Embodiments) Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. Fig. 1 is a diagram illustrating an overview of an image inspection device 100. Fig. 2 is a diagram illustrating an overview of an image inspection method. Note that, hereinafter, parts having common functions may be assigned the same reference numerals or symbols.

[0010] (Image Inspection Apparatus) The image inspection apparatus 100 according to the embodiment is used to inspect the appearance of products such as electronic components, for example.

[0011] 1, the image inspection device 100 includes a control unit S. The image inspection device 100 may include a camera C.

[0012] The camera C appropriately captures an image of the product (component) to be inspected. The captured image data (image data) is sent to the control means S. Note that image data captured by an external camera not included in the image inspection device 100 may be input to the control means S of the image inspection device 100, causing the image inspection device 100 to subsequently inspect the captured image data.

[0013] The control means S is a computer system including a CPU, a memory M (storage medium) such as a RAM, a ROM, and an auxiliary storage device, an external connection interface IF with an input / output device, and a bus B. The control means S may include a network interface. The memory M stores an image inspection program for causing the control means S (computer system) to execute the image inspection method according to this embodiment.

[0014] The CPU is an arithmetic circuit that controls the control means S. The CPU reads programs stored in the ROM or auxiliary storage device into the RAM. The CPU executes various processes in the programs read into the RAM. The ROM stores system programs and the like used to control the control means S. The auxiliary storage device stores application programs and the like that execute various processes. The auxiliary storage device is, for example, an HDD, an SSD, etc. The external connection interface IF is an interface for connecting various devices to the control means S. The external connection interface IF connects, for example, a camera C, a display D, a keyboard K, etc. to the control means S. The network interface functions to communicate via a network based on the control of the CPU. The bus B communicatively connects the above-mentioned functional units that make up the control means S.

[0015] The image inspection device 100 takes an image of the product to be inspected using a camera C, and based on the captured image data, a control means S determines whether the product to be inspected is pass / fail (whether the product is good or defective, normal or abnormal).

[0016] As shown in Figure 2, the image inspection device 100 performs inspections based on image data of a product. The image inspection device 100 performs a parallel combination of a defective product judgment to determine whether the product is good or defective and an abnormal product judgment to determine whether the product is normal or abnormal on the image data to be inspected. The image inspection device 100 then outputs an inspection result indicating OK (pass) if the product is good and normal, or NG (pass) if the product is defective or abnormal.

[0017] Defective product judgment is based on pass / fail rules learned from images of good and defective products collected in advance. When an unknown defective product image is input as an image to be inspected during the learning process, there is a possibility that an image that should actually be judged as defective will be mistakenly judged as good.

[0018] In order to reduce false positives in defective product detection, it is preferable to combine defective product detection with defective product detection and execute it in parallel. Defective product detection involves learning the likelihood of a product being good as a feature from only images of good products, and then calculating a defective product detection score for the image data of the inspection target. Defective product detection is performed based on whether the defective product detection score exceeds a threshold. This defective product detection score is normalized to a certain range. Specific methods for calculating the defective product detection score may include, for example, an algorithm using an autoencoder or an algorithm combining kernel density estimation and a projected histogram.

[0019] In this way, the image inspection device 100 performs a combination of defective product judgment and abnormal product judgment in parallel, thereby preventing erroneous judgment due to defective product judgment by abnormal product judgment, and enabling highly accurate inspection. Note that the image inspection device 100 may perform only abnormal product judgment.

[0020] However, algorithms using autoencoders may not produce stable anomaly detection scores for normal images. Furthermore, when the image data to be inspected is a color image, the anomaly detection scores may not be clearly separated between normal and abnormal images, resulting in inaccurate anomaly detection. Furthermore, when using a method that combines kernel density estimation and projected histograms, the anomaly detection scores must be properly normalized.

[0021] Therefore, the image inspection device 100 includes a training dataset acquisition unit 10 that acquires a training dataset including multiple training image data, a training feature derivation unit 20 that derives training features of the training image data, a clustering processing unit 30 that performs clustering of the training image data based on the training features, an index derivation unit 40 that derives, for each cluster obtained by clustering, the center of gravity of the multiple training features belonging to the cluster and the distance between the center of gravity and the training feature, and derives the average μ and variance σ of the multiple distances corresponding to the multiple training features, an imaging feature derivation unit 50 that extracts imaging features based on imaging data captured by a camera C, a minimum distance derivation unit 60 that derives each distance between the imaging feature and the multiple centers of gravity and derives a minimum distance X that is the smallest of the distances, an abnormality degree calculation unit 70 that calculates an abnormality degree α based on the minimum distance X, the average μ, and the variance σ, and a comparison unit 80 that compares the abnormality degree α with a preset threshold. As a result, even when the product to be inspected has multiple variations or the degree of variability in pass / fail products differs depending on the type of product, the abnormality degree α, which is an appropriately normalized evaluation score, can be calculated, making it possible to determine whether the product to be inspected is pass / fail based on a stable evaluation score with a clear threshold between pass / fail products.As a result, it is possible to provide an image inspection device 100 with high pass / fail determination accuracy.

[0022] The above-described components of the image inspection device 100 can be referred to as modules or units.

[0023] The abnormality level α is calculated using the following formula: α = {(X - μ) ２ / σ}. This allows the degree of anomaly α to be calculated as an appropriately normalized evaluation score, taking into account the minimum distance X, mean μ, and variance σ. This allows the degree of anomaly α, which is an evaluation score that is properly normalized, to be calculated. This allows the degree of variation in the appearance of the products being inspected, which differs depending on the type of product, to be absorbed, and the degree of anomaly α that the user sees to always be on the same scale.

[0024] The image inspection device 100 may include an error processing unit 90 that executes error processing when the abnormality level α exceeds a threshold. The error processing may involve, for example, displaying on a display that the inspection result of the inspected product is NG (abnormal or defective). This notifies the user that the product inspection result is NG.

[0025] The clustering processing unit 30 may perform clustering multiple times using random numbers. Then, the index derivation unit 40 may derive a center of gravity for each cluster each time clustering is performed multiple times, and use the representative cluster that has the smallest residual sum of squares of distances in the cluster as the population for calculating the minimum distance X. This makes it possible to derive a center of gravity that best captures the characteristics of the training dataset, even if the results change with each clustering attempt, thereby providing an image inspection device 100 with higher pass / fail judgment accuracy.

[0026] (Image Inspection Method) Next, the flow of the image inspection method will be described. The image inspection method can be performed using the image inspection device 100. Each step in the image inspection method can be performed by a corresponding unit in the image inspection device 100. Fig. 3 is a diagram illustrating the flow of the image inspection method.

[0027] (1) As shown in FIG. 3 , first, a training dataset including a plurality of training image data is acquired (training dataset acquisition step). The training image data are images of non-defective products. That is, the training dataset may be composed only of non-defective product images. The training dataset acquisition step can be executed by the training dataset acquisition unit 10 in the image inspection device 100.

[0028] (2) Next, learning features of the training image data acquired in the training dataset acquisition step (process) are derived (learning feature derivation step). The learning features may be derived using an appropriate algorithm based on the training image data. The algorithm for deriving the learning features may be, for example, a method using a set of brightness differences within an image as features (Haar-Like), a method using features focusing on the distribution of brightness gradient directions within a grayscale image (HOG), a method extracting features in two stages: feature point detection and feature description (SIFT), a filter method, a wrapper method, etc. The learning feature derivation step can be executed by the learning feature derivation unit 20 in the image inspection device 100.

[0029] Here, the learning feature derivation step may be executed according to the following flow. Note that the imaging feature derivation step described later may also be executed according to a flow similar to the flow of the learning feature derivation step shown here. Fig. 4 is a diagram illustrating the flow of the feature derivation step.

[0030] (2-1) As shown in Fig. 4, image data (learning image data or captured image data) may be resized. The resizing process is an operation for changing the size of image data, which is performed to unify the input size of a machine learning model.

[0031] (2-2) Next, the image data (learning image data or captured image data) that has been appropriately resized may be padded. The padding process is a process of adding meaningless data before or after short data to adjust its length so that the data is treated as a fixed length.

[0032] (2-3) Next, window function processing may be performed on the image data (learning image data or captured image data) that has been appropriately padded. Window function processing is a process of multiplying image data by a window function in order to extract only information within a finite interval.

[0033] (2-4) Next, FFT processing may be performed on the image data (learning image data or captured image data) that has been appropriately window function processed. The FFT processing is a process of Fourier transforming the image data to convert the spatial domain into the spatial frequency domain based on the image data and obtain an amplitude spectrum distribution.

[0034] (2-5) Next, band-pass filtering may be performed on the image data (learning image data or captured image data) that has been subjected to FFT processing as appropriate. Band-pass filtering is a process for compressing image data by passing only frequency components within a specific range, thereby removing components that have little effect on the judgment.

[0035] (3) Returning to FIG. 3 , next, based on the learning features derived in the learning feature derivation step, the learning image data is clustered and divided into multiple clusters (clustering processing step). Clustering is a machine learning technique that groups data based on the similarity between data. Clustering may be hierarchical clustering using algorithms such as shortest distance, longest distance, average distance, center of gravity, or Ward's method, or non-hierarchical clustering such as the k-means method. The algorithm used for clustering may be selected using random numbers each time the clustering process is performed. The algorithm used for clustering may also be changed using random numbers each time the clustering process is performed. Specifically, for example, one algorithm may be defined, and the random numbers used therein may be reassigned to perform clustering multiple times. The clustering processing step can be performed by the clustering processing unit 30 in the image inspection device 100.

[0036] (4) Next, for each cluster obtained in the clustering process, the center of gravity of the multiple learning features belonging to the cluster and the distance between the center of gravity and each learning feature are derived. Then, the mean μ and variance σ of the multiple distances derived for the multiple learning features are derived (index derivation step). The index derivation step can be performed by the index derivation unit 40 in the image inspection device 100. Through the above steps, a standard index for a non-defective product can be derived as pre-processing for inspection.

[0037] (5) Next, imaging features are extracted based on imaging data captured by the camera (imaging feature derivation step). The imaging data is image data obtained by capturing an image of the product to be inspected with a camera. The imaging features may be derived based on the imaging data using an appropriate algorithm. The extraction of the imaging features may be the same method as that used to derive the learning features in the learning feature derivation step. The imaging feature derivation step can be executed by the imaging feature derivation unit 50 in the image inspection device 100.

[0038] (6) Next, the distances between the image feature and the multiple centers of gravity are derived, and the minimum distance X, which is the smallest of the distances, is derived (minimum distance derivation step). The minimum distance derivation step can be executed by the minimum distance derivation unit 60 in the image inspection device 100. (7) Next, based on the minimum distance X derived in the minimum distance derivation step and the mean μ and variance σ derived in the index derivation step, specifically, a calculation formula (α={(X-μ) ２ / σ}) (abnormality degree calculation step). The abnormality degree calculation step can be executed by the abnormality degree calculation unit 70 in the image inspection device 100.

[0039] (8) Next, the degree of abnormality α calculated in the degree of abnormality calculation step is compared with a preset threshold (comparison step). The comparison step can be performed by the comparison unit 80 in the image inspection device 100.

[0040] As explained above, the image inspection method includes the steps described above. This makes it possible to stabilize the judgment score and clearly distinguish between good and bad products using a specific threshold. This improves the accuracy of the final pass / fail judgment of the product.

[0041] (9) After the comparison step, if the abnormality level α exceeds a threshold, an error process may be executed (error process step). The error process may be, for example, displaying on the display D a message indicating that the inspection result of the product being inspected is NG (abnormal or defective). This notifies the user that the product inspection result is NG. The error process step may be executed by the error processing unit 90 in the image inspection device 100.

[0042] (Image Inspection Program) The image inspection program has the function of causing a computer to execute each process performed in each step of the image inspection method described above. Specifically, the image inspection program has a learning dataset acquisition function, a learning feature derivation function, a clustering processing function, an index derivation function, an imaging feature derivation function, a minimum distance derivation function, an anomaly degree calculation function, and a comparison function, corresponding to each step of the image inspection method described above. This makes it possible to stabilize the judgment score and clearly distinguish between good and bad products using a specific threshold. This improves the accuracy of the final pass / fail judgment of the product.

[0043] The technical scope of the present invention is not limited to the above-described embodiments, and various modifications can be made without departing from the spirit of the present invention.

[0044] In addition, the components in the above-described embodiments may be replaced with known components as appropriate without departing from the spirit of the present invention. Furthermore, the above-described modifications may be combined as appropriate without departing from the spirit of the present invention.

[0045] As described above, the image inspection device 100 according to the embodiment includes: a training dataset acquisition unit 10 that acquires a training dataset including multiple pieces of training image data; a training feature derivation unit 20 that derives training features of the training image data; a clustering processing unit 30 that performs clustering of the training image data based on the training features; an index derivation unit 40 that derives, for each cluster obtained by clustering, the center of gravity of multiple training features belonging to the cluster and the distance between the center of gravity and the training feature, and derives the average μ and variance σ of multiple distances corresponding to the multiple training features; an imaging feature derivation unit 50 that extracts imaging features based on imaging data captured by a camera C; a minimum distance derivation unit 60 that derives each distance between the imaging feature and the multiple centers of gravity and derives the minimum distance X that is the smallest of the distances; an abnormality degree calculation unit 70 that calculates the abnormality degree α based on the minimum distance X, the average μ, and the variance σ; and a comparison unit 80 that compares the abnormality degree α with a preset threshold. This allows the calculation of the abnormality degree α, which is an appropriately normalized evaluation score, so that pass / fail judgment of the product to be inspected can be performed based on a stable evaluation score with a clear threshold value for determining whether a product is good or bad. Therefore, it is possible to provide an image inspection device 100 with high pass / fail judgment accuracy.

[0046] The image inspection method according to the embodiment includes a training dataset acquisition step of acquiring a training dataset including multiple training image data, a training feature derivation step of deriving training features from the training image data, a clustering processing step of clustering the training image data based on the training features to divide the training image data into multiple clusters, an index derivation step of deriving, for each cluster, the center of gravity of the multiple training features belonging to the cluster and the distances between the center of gravity and the training features, and deriving the mean μ and variance σ of the multiple distances corresponding to the multiple training features, an imaging feature derivation step of extracting imaging features based on imaging data captured by a camera, a minimum distance derivation step of deriving each distance between the imaging features and the multiple centers of gravity and deriving the smallest distance X among the distances, an anomaly degree calculation step of calculating an anomaly degree α based on the minimum distance X, the mean μ, and the variance σ, and a comparison step of comparing the anomaly degree α with a predetermined threshold. The anomaly degree α is calculated as an appropriately normalized evaluation score, enabling a pass / fail judgment of a product to be inspected based on a stable evaluation score with a clear threshold for distinguishing between good and bad products. Therefore, an image inspection method with high accuracy in pass / fail judgment can be provided.

[0047] According to an embodiment, an image inspection program causes a computer to execute the following functions: a training dataset acquisition function for acquiring a training dataset including multiple pieces of training image data; a training feature derivation function for deriving training features from the training image data; a clustering processing function for clustering the training image data based on the training features to divide the training image data into multiple clusters; an index derivation function for deriving, for each cluster, the center of gravity of the multiple training features belonging to the cluster and the distances between the center of gravity and the training features, and deriving the mean μ and variance σ of the multiple distances corresponding to the multiple training features; an imaging feature derivation function for extracting imaging features based on imaging data captured by a camera; a minimum distance derivation function for deriving each distance between the imaging features and the multiple centers of gravity and deriving the smallest distance X among the distances; an anomaly degree calculation function for calculating an anomaly degree α based on the minimum distance X, the mean μ, and the variance σ; and a comparison function for comparing the anomaly degree α with a preset threshold. The anomaly degree α is calculated as an appropriately normalized evaluation score, enabling pass / fail judgment of products to be inspected based on a stable evaluation score with a clear threshold for distinguishing between good and bad products. Therefore, an image inspection program with high pass / fail judgment accuracy can be provided.

[0048] 100 Image inspection device, 10 Learning dataset acquisition unit, 20 Learning feature derivation unit, 30 Clustering processing unit, 40 Index derivation unit, 50 Imaging feature derivation unit, 60 Minimum distance derivation unit, 70 Anomaly degree calculation unit, 80 Comparison unit, 90 Error processing unit, α Anomaly degree, μ Mean, σ Variance, B Bus, C Camera, D Display, K Keyboard, M Memory, S Control means, X Minimum distance

Claims

1. An image inspection device comprising: a training dataset acquisition unit that acquires a training dataset including a plurality of training image data; a training feature derivation unit that derives training features of the training image data; a clustering processing unit that clusters the training image data based on the training features to divide it into a plurality of clusters; an index derivation unit that derives, for each cluster, the center of gravity of the plurality of training features belonging to the cluster and the distance between the center of gravity and the training feature, and derives an average μ and variance σ of the plurality of distances corresponding to the plurality of training features; an imaging feature derivation unit that extracts imaging features based on imaging data captured by a camera; a minimum distance derivation unit that derives each distance between the imaging feature and the plurality of centers of gravity and derives a minimum distance X that is the smallest of the distances; an abnormality degree calculation unit that calculates an abnormality degree α based on the minimum distance X, the average μ, and the variance σ; and a comparison unit that compares the abnormality degree α with a preset threshold.

2. The image inspection device according to claim 1, further comprising an error processing unit that executes error processing when the abnormality level α exceeds the threshold value.

3. The image inspection device according to claim 1 or claim 2, wherein the clustering processing unit performs the clustering multiple times using random numbers, and the index derivation unit derives the center of gravity for each cluster each time the clustering is performed, and a representative cluster for which the residual sum of squares of the distances in the clusters is smallest is used as a population for calculating the minimum distance X.

4. An image inspection device according to claim 1 or claim 2, wherein the imaging feature derivation unit or the learning feature derivation unit performs resizing processing, window function processing, FFT processing or band-pass filter processing.

5. An image inspection method comprising: a training dataset acquisition step of acquiring a training dataset including a plurality of training image data; a training feature derivation step of deriving training features of the training image data; a clustering processing step of clustering the training image data based on the training features to divide it into a plurality of clusters; an index derivation step of deriving, for each cluster, the center of gravity of the plurality of training features belonging to the cluster and the distance between the center of gravity and the training feature, and deriving an average μ and variance σ of the plurality of distances corresponding to the plurality of training features; an imaging feature derivation step of extracting imaging features based on imaging data captured by a camera; a minimum distance derivation step of deriving each distance between the imaging feature and the plurality of centers of gravity and deriving a minimum distance X that is the smallest of the distances; an abnormality degree calculation step of calculating an abnormality degree α based on the minimum distance X, the average μ, and the variance σ; and a comparison step of comparing the abnormality degree α with a predetermined threshold.

6. An image inspection program that causes a computer to execute: a training dataset acquisition function that acquires a training dataset including a plurality of training image data; a training feature derivation function that derives training features of the training image data; a clustering processing function that clusters the training image data based on the training features to divide it into a plurality of clusters; an index derivation function that derives, for each cluster, the center of gravity of the plurality of training features belonging to the cluster and the distance between the center of gravity and the training feature, and derives the average μ and variance σ of the plurality of distances corresponding to the plurality of training features; an imaging feature derivation function that extracts imaging features based on imaging data captured by a camera; a minimum distance derivation function that derives each distance between the imaging feature and the plurality of centers of gravity and derives the minimum distance X that is the smallest of the distances; an abnormality degree calculation function that calculates an abnormality degree α based on the minimum distance X, the average μ, and the variance σ; and a comparison function that compares the abnormality degree α with a preset threshold.

Images