Apparatus for visual inspection and method for visual inspection

Abstract

To provide an appearance inspection device and an appearance inspection method which enable fast learning, are easy to use, have a high abnormality detection performance of a wide size range, and also have a high robust performance.SOLUTION: A processor: inputs a non-defective article image to a convolution neural network; outputs a feature map having a feature of the non-defective article image, from each hierarchy of the convolution neural network; and generates a plurality of models for detecting abnormality for each hierarchy, on the basis of the feature map. The processor executes inspection processing of: inputting an inspection object image to the convolution neural network; outputting the feature map having the feature of the inspection object image, from each hierarchy of the convolution neural network; applying the feature map to a learned model; outputting an abnormal level map indicating an abnormal level for each hierarchy; and determining presence / absence of the abnormality of the inspection object image, on the basis of the plurality of outputted abnormal level maps.SELECTED DRAWING: Figure 5

Description

Technical Field

[0001] The present invention relates to an appearance inspection apparatus and an appearance inspection method for inspecting the appearance of a workpiece.

Background Art

[0002] For example, Patent Document 1 discloses a processing device that uses machine learning by a computer to determine whether a workpiece is a non-defective product or a defective product. The processing device of Patent Document 1 performs supervised machine learning on non-defective product data to generate a non-defective product learning model, and performs supervised machine learning on defective product data to generate a defective product learning model. After that, the data of the workpiece to be determined is input, and it is configured to be able to determine whether the workpiece is a non-defective product or a defective product by the non-defective product learning model and the defective product learning model. Such a device is also called a workpiece appearance inspection device.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] By the way, in the production site of workpieces, defective products hardly occur, so it is easy to collect a large amount of non-defective product data, while it is difficult to collect defective product data. Therefore, supervised learning for non-defective product data is assumed as a solution, but in the case of a machine learning network trained only with non-defective product data, there are the following problems.

[0005] In other words, there are three methods for learning good images. The first method is called TEXEMS (Texture Exemplars for Defect Detection on Random Textured Surfaces). In this method, good image data is divided into multiple patches during training, and a Gaussian Mixture Model (GMM) is used to model the patches. During inference, the input image is divided into patches, and the anomaly score of each patch in the input image is calculated using the GMM generated during training, and an anomaly score map is generated. In this method, the modeling target is patches (raw pixels), so the level of abstraction is not sufficient, resulting in low detection performance and robustness.

[0006] The second method is called DHM (Deep Hybrid Models). In this method, training images are input into a neural network during training, feature extraction is performed, and the extracted features are modeled using anomaly detection methods such as OC-SVM (One Class Support Vector Machine) or SVDD (support vector data description). During inference, the input image is input into the neural network for feature extraction, and the extracted features can be classified using anomaly detection methods such as OC-SVM or SVDD that were generated during training. This method improves detection performance and robustness compared to the first method by using high-level abstraction features extracted by the neural network, but it can only detect anomalies that represent the entire input image, making it difficult to detect a wide range of anomalies, including localized, fine anomalies like those found in visual inspections.

[0007] A third method involves using an autoencoder (AutoEncoder) or DAE (Denoising AutoEncoder). In this method, the model is trained to reconstruct noisy, good images back into their original images. During inference, a deep neural network (DNN) generates a reconstructed image from the input image and creates an anomaly map (an image where pixel values ​​represent the degree of anomaly in the corresponding area) showing the difference between the input image and the reconstructed image. This method requires time to train the neural network and is time-consuming to adjust the parameters. Furthermore, even in good images, large differences can occur in areas with significant variations, such as areas with large lighting fluctuations, metallic patterns, and near edges, potentially leading to over-detection. Additionally, because autoencoders and DAEs have high generalization capabilities, they may reconstruct anomalies as well, potentially leading to missed detections.

[0008] This disclosure is made in view of the above points, and its purpose is to provide an appearance inspection device and an appearance inspection method that enable high-speed learning, are easy to use, have high anomaly detection performance across a wide range of sizes, and are highly robust. [Means for solving the problem]

[0009] To achieve the above objective, one aspect of this disclosure may provide a visual inspection device comprising a processor and memory for constructing a convolutional neural network having convolutional layers and a modeling means for generating a model for detecting anomalies based on feature maps output from the neural network, and using the convolutional neural network and the modeling means to detect anomalies contained in an image to be inspected. The processor inputs images of good products corresponding to good products as training images into the convolutional neural network, causes each layer of the convolutional neural network to output feature maps having the characteristics of the good product images, and executes a learning process in which the modeling means generates a plurality of models for detecting anomalies for each layer based on the feature maps output from each layer. Furthermore, the processor inputs the image to be inspected into the convolutional neural network, outputs a feature map containing the characteristics of the image to be inspected from each layer of the convolutional neural network, applies the feature maps output from each layer to the model generated by the learning process, outputs an anomaly degree map indicating the degree of anomaly for each layer, and performs an inspection process to detect anomalies contained in the image to be inspected based on the multiple anomaly degree maps output.

[0010] In other words, the feature maps output from each layer of the convolutional neural network during the training process are such that those output from higher layers are more abstract and represent wider-area anomalies, while those output from lower layers are less abstract and represent more localized anomalies. Because the model is generated based on these feature maps with multiple different characteristics, it achieves high performance in detecting anomalies across a wide range of sizes, is less prone to false positives and false negatives, and is also a highly robust model. Furthermore, compared to the third method mentioned above, which generates reconstructed images using a deep neural network, the training time is shortened, and it is easier for users to use because it does not require the effort of parameter tuning.

[0011] Another modeling means is composed of an autoencoder and includes an encoding layer for compressing the feature map and a decoding layer for restoring the compressed feature map.

[0012] This configuration allows for high-speed and high-performance detection by utilizing an autoencoder.

[0013] In other embodiments of convolutional neural networks, the number of dimensions of the multidimensional vectors at the local locations of the feature map increases as one progresses from lower to higher layers.

[0014] This configuration allows for an expansion of the coverage area on good product images where features can be represented at local locations within the feature map.

[0015] An appearance inspection apparatus according to another embodiment may further include a storage unit for storing verification images for verifying the abnormality detection performance of the model generated by the modeling means, and a display control unit for controlling a display unit. The processor outputs an abnormality map corresponding to the verification image for each layer using a plurality of models based on the verification image read from the storage unit, and calculates the degree of separation between good / bad images and defective images of the verification image at each layer based on the plurality of output abnormality maps or a composite abnormality map obtained by combining the plurality of abnormality maps.

[0016] Furthermore, the display control unit can generate a user interface screen capable of displaying the degree of separation at each level calculated by the processor and the anomaly map at each level, and can display the generated user interface screen on the display unit.

[0017] In this configuration, pre-annotated validation images are input into a convolutional neural network, and each model outputs an anomaly score map for each layer based on the validation images. Using these anomaly score maps, the degree of separation of the validation images at each layer can be calculated. The anomaly score maps output from models at layers with a separation score above a certain level can be incorporated into the user interface screen and displayed for the user. By checking the anomaly score maps at layers with a separation score above a certain level, the user can easily determine whether the training state is ready for deployment.

[0018] A processor in another embodiment is configured to select the anomaly score map or composite anomaly score map output from the model of the highest hierarchy in which the calculated separation score is greatest.

[0019] With this configuration, by examining the anomaly score map or composite anomaly score map of the hierarchical level with the highest degree of separation, it is possible to accurately determine whether the learned state is suitable for deployment.

[0020] In other embodiments, the system further includes an adjustment acceptance unit that accepts adjustments to the threshold by the user. For example, there may be layers where over-detection occurs even with a high degree of separation, and the user can check these layers.

[0021] A processor in another embodiment is configured to detect abnormalities in the image to be inspected based on a plurality of abnormality maps output from each model, and to determine whether the verification image is a good product image or a defective product image based on the detected abnormalities. The display control unit can generate a cumulative histogram based on the frequency at which the verification image is determined to be a good product image and the frequency at which the verification image is determined to be a defective product image, and display it on the display unit. The adjustment reception unit is configured to accept adjustments to the threshold on the cumulative histogram displayed on the user interface screen.

[0022] According to this configuration, the abnormality maps of each layer can be confirmed at once, and the comparison between the abnormality maps can be easily performed.

[0023] A processor according to another aspect generates a combined abnormality map obtained by combining a plurality of the abnormality maps, calculates the separation degree of the verification image based on the generated combined abnormality map or the abnormality map output from the model of each layer, and selects the combined abnormality map when the separation degree calculated based on the combined abnormality map is a predetermined value or more. The display control unit can generate a user interface screen capable of displaying the combined abnormality map selected by the processor, and cause the display unit to display the generated user interface screen.

[0024] According to this configuration, the separation degree based on the abnormality map can be displayed in an easy-to-understand manner in the form of a cumulative histogram.

[0025] In another aspect, it further includes an input unit that receives an input by a user as to whether to start the learning adjustment mode. When the input unit receives the start of the learning adjustment mode, the processor can select the abnormality map corresponding to the verification image of each layer or the combined abnormality map. The display control unit generates a user interface screen having a list display area capable of displaying, in a list format, the abnormality maps corresponding to the verification images of each layer selected by the processor, and causes the display unit to display the generated user interface screen.

[0026] According to this configuration, by reflecting the adjustment of the threshold value, the abnormality map of an appropriate layer can be selected.

Effect of the Invention

[0027] As explained above, by inputting images of good products as training images into a convolutional neural network, feature maps are output from each layer, and multiple models for detecting anomalies are generated for each layer based on the output feature maps. This enables high-speed learning, ease of use, improved anomaly detection performance across a wide range of sizes, and increased robustness. [Brief explanation of the drawing]

[0028] [Figure 1] This is a schematic diagram showing the configuration of an appearance inspection device according to an embodiment of the present invention. [Figure 2] This block shows the hardware configuration of the aforementioned visual inspection device. [Figure 3] This is a flowchart illustrating an example of the learning process. [Figure 4] This figure shows an example of a user interface screen displayed on a display device. [Figure 5] This diagram illustrates how to generate multiple models. [Figure 6] This diagram illustrates how to generate an anomaly score map after training. [Figure 7] This figure shows an example of a user interface screen displaying an anomaly severity map that appears natural. [Figure 8] This figure shows an example of a user interface screen displaying an anomaly severity map that appears abnormal due to over-detection or other issues. [Figure 9] This figure shows an example of a user interface screen displaying anomaly maps and isolation graphs for each hierarchical level. [Figure 10] This figure shows an example of a user interface screen where a specific anomaly severity map has been selected by the user. [Figure 11] This figure shows an example of the user interface screen displayed after tuning. [Figure 12] This is a diagram equivalent to Figure 9, showing the case where false positives occur at all levels. [Figure 13]This is a diagram equivalent to Figure 10, showing the case where false positives occur at all levels. [Figure 14] This is a diagram equivalent to Figure 11, showing the case where false positives occur at all levels. [Figure 15] This flowchart shows an example of the procedure for operating a visual inspection device. [Figure 16] This diagram illustrates an example of modeling during training. [Figure 17] This diagram illustrates an example of modeling during operation. [Figure 18] This diagram illustrates an example of how to speed up processing. [Modes for carrying out the invention]

[0029] Embodiments of the present invention will be described in detail below with reference to the drawings. The following description of preferred embodiments is essentially illustrative and is not intended to limit the present invention, its applications, or its uses.

[0030] Figure 1 is a schematic diagram showing the configuration of a visual inspection device 1 according to an embodiment of the present invention. The visual inspection device 1 is a device for determining the quality of a workpiece image (also called an inspection target image) acquired by imaging a workpiece that is the object of inspection, such as various parts or products, and can be used in production sites such as factories. Specifically, the visual inspection device 1 has a convolutional neural network having a convolutional layer and a modeling means for detecting abnormalities based on a feature map output from the neural network. The visual inspection device 1 is configured to detect abnormalities contained in the inspection target image using the convolutional neural network and the modeling means.

[0031] A workpiece may be inspected as a whole, or only a part of it may be inspected. Furthermore, a single workpiece may contain multiple items to be inspected. Also, a workpiece image may contain multiple workpieces.

[0032] The visual inspection device 1 comprises a control unit 2, which serves as the main body of the device; an imaging unit 3; a display device (display unit) 4; and a personal computer 5. The personal computer 5 is not mandatory and can be omitted. Various information and images can be displayed using the personal computer 5 instead of the display device 4, or the functions of the personal computer 5 can be incorporated into the control unit 2 or the display device 4.

[0033] Figure 1 shows an example of the configuration of the visual inspection device 1, consisting of a control unit 2, an imaging unit 3, a display device 4, and a personal computer 5. However, any multiple of these components can be combined and integrated. For example, the control unit 2 and the imaging unit 3 can be integrated, or the control unit 2 and the display device 4 can be integrated. Furthermore, the control unit 2 can be divided into multiple units and some parts incorporated into the imaging unit 3 or the display device 4, or the imaging unit 3 can be divided into multiple units and some parts incorporated into other units.

[0034] (Configuration of imaging unit 3) As shown in Figure 2, the imaging unit 3 comprises a camera module (imaging unit) 14 and an illumination module (illumination unit) 15, and is a unit that performs the acquisition of workpiece images. The camera module 14 comprises an AF motor 141 that drives the imaging optical system and an imaging substrate 142. The AF motor 141 is the part that automatically adjusts the focus by driving the lens of the imaging optical system, and can perform focus adjustment using conventional methods such as contrast autofocus. The imaging substrate 142 is equipped with a CMOS sensor 143 as a light-receiving element that receives light incident from the imaging optical system. The CMOS sensor 143 is an imaging sensor configured to acquire color images. Instead of the CMOS sensor 143, a light-receiving element such as a CCD sensor can also be used.

[0035] The lighting module 15 includes an LED (light-emitting diode) 151 as a light-emitting element that illuminates the imaging area including the workpiece, and an LED driver 152 that controls the LED 151. The timing of light emission, the duration of light emission, and the amount of light emitted by the LED 151 can be arbitrarily controlled by the LED driver 152. The LED 151 may be provided integrally with the imaging unit 3, or it may be provided as a separate external lighting unit.

[0036] (Configuration of display device 4) The display device 4 has a display panel made of, for example, a liquid crystal panel or an organic EL panel. Work images, user interface images, etc., output from the control unit 2 are displayed on the display device 4. In addition, if the personal computer 5 has a display panel, the display panel of the personal computer 5 can be used as a substitute for the display device 4.

[0037] (Operation equipment) Examples of operating devices for a user to operate the visual inspection device 1 include the keyboard 51 and mouse 52 of a personal computer 5, but are not limited to these; any device configured to accept various operations by the user is acceptable. For example, a pointing device such as the touch panel 41 of a display device 4 is also included as an operating device.

[0038] User operations on the keyboard 51 and mouse 52 can be detected by the control unit 2. The touch panel 41 is a conventional touch-type control panel equipped with, for example, a pressure sensor, and user touch operations can be detected by the control unit 2. The same applies when other pointing devices are used.

[0039] (Configuration of Control Unit 2) The control unit 2 comprises a main board 13, a connector board 16, a communication board 17, and a power supply board 18. The main board 13 is equipped with a processor 13a. The processor 13a controls the operation of each connected board and module. For example, the processor 13a outputs a lighting control signal to the LED driver 152 of the lighting module 15 to control the on / off state of the LED 151. The LED driver 152 switches the on / off state of the LED 151 and adjusts the on / off time, as well as the light intensity of the LED 151, in response to the lighting control signal from the processor 13a.

[0040] Furthermore, the processor 13a outputs an imaging control signal to the imaging substrate 142 of the camera module 14 to control the CMOS sensor 143. The CMOS sensor 143 starts imaging in response to the imaging control signal from the processor 13a and adjusts the exposure time to an arbitrary time to perform imaging. In other words, the imaging unit 3 images the area within the field of view of the CMOS sensor 143 in response to the imaging control signal output from the processor 13a. If there is a workpiece within the field of view, the workpiece will be imaged, but if there are objects other than workpieces within the field of view, those can also be imaged. For example, the visual inspection device 1 can use the imaging unit 3 to capture images of good products corresponding to good products as training images for a convolutional neural network. The training images do not have to be images captured by the imaging unit 3; they may be images captured by other cameras, etc.

[0041] On the other hand, when the visual inspection device 1 is in operation, the imaging unit 3 can capture images of the workpiece. Furthermore, the CMOS sensor 143 is configured to output live images, that is, the currently captured images, in real time at a short frame rate.

[0042] Once imaging by the CMOS sensor 143 is complete, the image signal output from the imaging unit 3 is input to the processor 13a on the main board 13 for processing and stored in the memory 13b of the main board 13. Details of the specific processing performed by the processor 13a on the main board 13 will be described later. The main board 13 may also be equipped with processing devices such as FPGAs and DSPs. Alternatively, the processor 13a may be an integrated processor with FPGAs and DSPs.

[0043] The connector board 16 is the part that receives power from the outside via a power connector (not shown) provided on the power interface 161. The power supply board 18 is the part that distributes the power received by the connector board 16 to each board and module, and specifically distributes power to the lighting module 15, camera module 14, main board 13, and communication board 17. The power supply board 18 is equipped with an AF motor driver 181. The AF motor driver 181 supplies drive power to the AF motor 141 of the camera module 14, thereby realizing autofocus. The AF motor driver 181 adjusts the power supplied to the AF motor 141 according to the AF control signal from the processor 13a of the main board 13.

[0044] The communication board 17 is the part that performs communication between the main board 13 and the display device 4 and the personal computer 5, and communication between the main board 13 and external control equipment (not shown). Examples of external control equipment include programmable logic controllers. Communication may be wired or wireless, and either communication method can be implemented using conventionally known communication modules.

[0045] The control unit 2 is provided with a storage device (storage unit) 19, which may consist of, for example, a solid-state drive or a hard disk drive. The storage device 19 stores program files 80 and configuration files (software) that enable the hardware to execute the various controls and processes described later. The program files 80 and configuration files can be stored on a storage medium 90, such as an optical disc, and the program files 80 and configuration files stored on this storage medium 90 can be installed on the control unit 2. The program files 80 may also be downloaded from an external server using a communication line. The storage device 19 can also store, for example, the image data, parameters for constructing the convolutional neural network of the visual inspection device 1, modeling means and parameters for constructing a model using said modeling means.

[0046] Specifically, the processor 13a and memory 13b of the visual inspection device 1 read parameters stored in, for example, a storage device 19, and construct a model using a convolutional neural network and modeling means. A work image of the work to be inspected is input to the constructed convolutional neural network, and feature maps containing the characteristics of the work image are output from each layer of the convolutional neural network. The feature maps output from each layer are applied to a trained model, and an abnormality map indicating the degree of abnormality is output for each layer. The device is configured to detect abnormalities contained in the work image based on the multiple abnormality maps that are output. By using this visual inspection device 1, a visual inspection method that determines the quality of a work based on a work image can be performed.

[0047] Furthermore, as shown in Figure 2, the processor 13a is provided with an input unit 13d and an adjustment reception unit 13e. The input unit 13d and the adjustment reception unit 13e may be hardware components or components that are executed by software. Also, the input unit 13d and the adjustment reception unit 13e do not necessarily have to be provided on the main board 13; some or all of them may be provided on boards other than the main board 13.

[0048] The learning process will be explained in detail below based on the flowchart shown in Figure 3. This process is performed before the operation of the visual inspection device 1 begins, and can therefore be called the startup process. This flowchart can be started, for example, at the same time as the startup process begins. Before starting, the good product images are stored in the storage device 19 as learning images. The storage device 19 also stores verification images to verify the anomaly detection performance of the model generated by the modeling means. The verification images may consist only of defective product images, only of good product images, or both defective and good product images. The verification images may be images acquired before the learning process, images newly acquired for the learning process, or images acquired during the operation of the visual inspection device 1. If the verification image is a defective product image, annotation is performed on the defective product image. That is, the user performs in advance processing such as indicating that the defective product image is a defective product image, or specifying the defective part of the defective product image. Note that specifying the defective part is not always necessary. The information added by the annotation is stored in the storage device 19 in association with the corresponding defective product image.

[0049] In step SA1 after the start, the system accepts learning operations from the user. That is, by accepting learning operations from the user, the following processes are executed in order, and the system waits without executing any processes until a learning operation is accepted. During the learning process, the display control unit 13c first generates a user interface screen 200 as shown in Figure 4 and displays it on the display device 4. The user interface screen 200 is provided with a learning image display area 201, a verification image display area 202, an abnormality map display area 203, a separation degree display area 204, and an operation button display area 205. The learning image display area 201 displays a learning image 300 read from the storage device 19. The learning image 300 can be an image arbitrarily selected by the user from among the good product images stored in the storage device 19. For example, after the user checks the good product images stored in the storage device 19, the processor 13a detects the good product image selected by the user using the mouse 52, etc., and the display control unit 13c can incorporate the good product image detected by the processor 13a into the learning image display area 201 and display it.

[0050] The verification image display area 202 displays verification images 301 and 302 read from the storage device 19. Verification image 301 is an image of a good product, and verification image 302 is an image of a defective product. The black dot in verification image 302 is the defective part Wa of the workpiece W. Verification images 301 and 302, like the training image 300, can also be images arbitrarily selected by the user from the verification images stored in the storage device 19.

[0051] The operation button display area 205 is provided with a learning button 205a, an inference button 205b, and a debug button 205c. These buttons 205a, 205b, and 205c can be operated by the user, and the operation of buttons 205a, 205b, and 205c is received by the input unit 13d.

[0052] When the input unit 13d detects that the learning button 205a has been operated, the process proceeds from step SA1 to step SA2. Step SA2 is the step in which model learning is performed using training images, and this step corresponds to the learning step. The processing performed in step SA2 will be explained with reference to Figure 5. First, a pre-trained convolutional neural network 70 is prepared. For example, an image classification network that has already been trained to extract strong and informative features from natural images is constructed as the pre-trained convolutional neural network 70 using the processor 13a and memory 13b.

[0053] Processor 13a inputs training images to the convolutional neural network 70. By using the convolutional neural network 70, features can be hierarchically convolved to construct a feature pyramid. The corresponding local features for each position in the feature map of each level (hereinafter also referred to as a hierarchy) are the corresponding channel (number of dimensions) direction vectors for that position. The channel direction corresponds to the left-right direction in Figure 5, and the information aggregated in this channel direction vector corresponds to the receptive field on the original image centered on the spatial position of the vector. The corresponding receptive field of a feature becomes larger as the hierarchy of the convolutional neural network 70 increases.

[0054] In this process, the output feature maps of the final Conv Blocks of each layer of the convolutional neural network 70 are modeled using each (convolutional) autoencoder 71 and 72. Training is performed so that the output of each autoencoder 71 and 72 matches the input feature map. Only autoencoders 71 and 72 are trained. Autoencoders 71 and 72 are constructed and modeled for each layer. The modeling method used in this process consists of an autoencoder. The autoencoder has an encoding layer that compresses the number of Channel 1 dimensions of the feature map and a decoding layer that restores the compressed feature map. By directly modeling the feature maps of each layer with convolutional autoencoders, it is possible to balance the features within a single image, leading to improved learning stability and detection robustness. Furthermore, when using convolutional autoencoders with a fixed kernel size, the relationships between feature vectors at adjacent positions in the spatial direction on the feature map are utilized, leading to improved detection performance and robustness.

[0055] Figure 5 shows an example of modeling feature maps at two levels, illustrating autoencoders 71 and 72 at levels 5 (level 5) and 3 (level 3). In reality, autoencoders at other levels also exist. As shown in this figure, the number of dimensions of the multidimensional vectors at the local locations of the feature maps increases as you progress from lower to higher levels in the convolutional neural network 70. Specifically, it is 256 dimensions at level 3, but 512 dimensions at level 5. Thus, when a training image is input to the convolutional neural network 70, feature maps containing the features of the training image can be output from each level of the convolutional neural network 70. Based on the feature maps output from each level of the convolutional neural network 70, multiple models for detecting anomalies can be generated for each level by the modeling means.

[0056] For example, in the case of level 5, a corresponding feature map (spatial size: 14x14, number of channels: 512) is generated for one image and used as input to the autoencoder 71. This leads to faster learning and improved performance.

[0057] The Loss function can be defined as follows: First, feature extraction is performed on the training images using a convolutional neural network (Pre-Trained NN). The convolutional neural network outputs a tuple containing the feature maps of each layer.

[0058]

number

[0059] An autoencoder needs to be constructed for each output feature at each level. For a single feature map, the autoencoder's loss definition is as follows. As an example of a loss definition method, we use MSE (Mean Squared Error).

[0060]

number

[0061] x and y are the horizontal and vertical coordinates of the feature map in Spatia space, respectively.

[0062] c is the channel direction position of the feature map.

[0063] n is the number of elements in the feature map.

[0064] After model training, the process proceeds to step SA3, where processor 13a inputs the validation image into the convolutional neural network 70. Feature maps containing the features of the validation image are output from each layer of the convolutional neural network 70. The feature maps output from each layer are applied to the model generated in step SA2, and an anomaly score map indicating the degree of anomaly is output for each layer. In this step SA4, the anomaly score maps output from each layer are combined to generate a combined anomaly score map.

[0065] An example is shown in Figure 6. This figure shows Layer 5 (512 dimensions). First, the pre-trained neural network 70 used during learning extracts features from each layer and outputs a feature map. The output feature map is input to the trained autoencoder 71 of the corresponding layer, and inference processing is performed. The autoencoder 71 calculates the square of the norm of the difference between the corresponding local features at each position in the input and output feature maps, and generates an anomaly score map as the anomaly score of the corresponding position. At this time, after the anomaly score map for each layer is obtained, it is upsampled to the input image size and a synthesis process is performed. The synthesis process applies an AND operation to the anomaly score maps of adjacent layers on a pixel-by-pixel basis, and the obtained result is used to take the average on a pixel-by-pixel basis.

[0066] After the processor 13a generates an anomaly score map for each hierarchical level and a composite anomaly score map, the process proceeds to step SA5. In step SA5, the processor 13a calculates the degree of separation of the verification image at each hierarchical level based on the anomaly score map for each hierarchical level. The method for calculating the degree of separation Cn from the verification image at each hierarchical level based on the anomaly score map Sn (including the composite anomaly score map) is as follows.

[0067] 1. The image anomaly score ISn at each hierarchical level n of image I is the mean of the extracted anomalies in the corresponding anomaly severity map Sn (mean of pixel values).

[0068]

number

[0069] Here, n is the number of pixels in the anomaly map S, and x and y are the pixel coordinates of the anomaly map S.

[0070] As an example, the average pixel value of the anomaly score map is used, but the maximum anomaly on the anomaly score map Sn could also be used as the image anomaly score ISn for the corresponding hierarchical level. 2. For each hierarchical level n, calculate the resolution Cn based on the corresponding image anomaly score ISn. There are various methods for calculating the resolution Cn, but as an example, AUC-ROC (a larger value indicates a higher resolution) can be used. If the resolution based on AUC-ROC is the same, the resolution may be further fine-tuned by considering the anomaly detection intensity of the entire test image for the corresponding hierarchical level n and using the difference in anomaly detection intensity (average anomaly score of anomaly images - average anomaly score of normal images).

[0071] Alternatively, the corresponding resolution Cn of the smaller difference in anomaly detection intensity may be reduced. The difference in anomaly detection intensity SDn for hierarchical level n can be calculated using the following formula.

[0072]

number

[0073] Here, m is the number of abnormal images and l is the number of normal images.

[0074] Furthermore, the same calculation method can be used for the separation of verification images based on the composite anomaly map.

[0075] After calculating the separation degree of the verification images and the separation degree based on the composite anomaly map at each hierarchical level as described above, the process proceeds to step SA6. In step SA6, the processor 13a selects the anomaly map output from the model at each hierarchical level based on the thresholds used in step SA5 to determine whether the verification images at each hierarchical level are good or bad images. Specifically, the processor 13a is configured to select the anomaly map output from the model at the hierarchical level with the highest calculated separation degree. Similarly, the processor 13a is configured to compare the separation degree calculated based on the composite anomaly map with the threshold and select the composite anomaly map output from the model at the hierarchical level with the highest separation degree.

[0076] After the processor 13a selects an anomaly map, the process proceeds to step SA7. In step SA7, the display control unit 13c generates a user interface screen 200 (shown in Figure 7) capable of displaying the anomaly map selected by the processor 13a, and displays the generated user interface screen 200 on the display device 4. When the user selects one or both of the verification images 301 and 302 displayed in the verification image display area 202 of the user interface screen 200 by operating the mouse 52 or the like, selection frames 301a and 302a surrounding the selected images are displayed on the user interface screen 200.

[0077] The processor 13a selects either the anomaly map 311 or a composite anomaly map output from the model with the highest separation degree among the anomaly maps corresponding to the selected verification image 301. The display control unit 13c then displays the anomaly map 311 selected by the processor 13a in the anomaly map display area 203. If a verification image 302 consisting of images of defective products is selected, the anomaly map 312 output from the model with the highest separation degree among the anomaly maps corresponding to that verification image 302 is displayed in the anomaly map display area 203.

[0078] The display control unit 13c displays the separation degree in the separation degree display area 204 of the user interface screen 200. In this example, a cumulative histogram is displayed in the separation degree display area 204. Specifically, the processor 13a is configured to detect anomalies in the input image (verification image during verification, inspection target image during operation) based on multiple anomaly maps output from each model, and to determine whether the input image is a good product image or a defective product image based on the detected anomalies. The display control unit 13c generates a cumulative histogram 320 based on the frequency at which the verification image was determined to be a good product image and the frequency at which the verification image was determined to be a defective product image, and incorporates the generated cumulative histogram 320 into the separation degree display area 204 and displays it on the display device 4.

[0079] Next, the user proceeds to step SA8 to decide whether or not to switch to debug mode (learning adjustment mode). Debug mode is the mode to be switched to if there is something unusual about the abnormality maps 311 and 312 displayed in step SA7. The example shown in Figure 7 is an example where the user determines that there is nothing unusual and no over-detection. In other words, there is no reaction in abnormality map 311 corresponding to verification image 301 consisting of images of good products, and in abnormality map 312 corresponding to verification image 302 consisting of images of defective products, a white dot is seen as a reaction only at the defective area Wa, and there is no over-detection.

[0080] As shown in Figure 7, if the user determines that there are no abnormalities in the anomaly maps 311 and 312, the user operates the inference button 205b in step SA8. The input unit 13d detects that the inference button 205b has been operated. The operation of the inference button 205b means that debug mode will not be started, and the input unit 13d receives input from the user that debug mode will not be started. When the processor 13c detects that the input unit 13d has received input that debug mode will not be started, it determines that step SA8 was NO and terminates the learning process.

[0081] On the other hand, the example shown in Figure 8 is one in which the user judges something to be amiss. It is the same as the example shown in Figure 7 in that there is no reaction in the abnormality map 311 corresponding to the verification image 301 consisting of images of good products. However, the abnormality map 312 corresponding to the verification image 302 consisting of images of defective products shows reactions in addition to the defective area Wa. In the abnormality map 312 shown in Figure 8, in addition to the white dot corresponding to the defective area Wa, multiple small white dots and a white frame corresponding to the outer shape of the workpiece W are also displayed. The multiple small white dots and white frame are parts that reacted even though they are not defective areas. For example, due to variations in the background, over-detection occurs not only in the defective area Wa but also in the pattern and contour of the workpiece W. Therefore, this indicates that the current model needs to be adjusted.

[0082] As shown in Figure 8, if the user determines that there is something unusual in the abnormality maps 311 and 312, the user operates the debug button 205c in step SA8. When the input unit 13d detects that the debug button 205c has been operated, it receives input from the user indicating whether or not to start debug mode. In other words, the input unit 13d is the part that receives input from the user indicating whether or not to start debug mode.

[0083] When the input unit 13d receives a signal to start debug mode, the processor 13a determines that step SA8 is YES and proceeds to step SA9. In step SA9, the abnormality maps for each level are displayed on the display device 4. Figure 9 shows the user interface screen 200 that the display control unit 13c displays on the display device 4 when the system enters debug mode. The abnormality map display area 203 of the user interface screen 200 is a list display area that can display multiple abnormality maps 311 and 312 in a list format. The display format of the abnormality maps 311 and 312 is not limited to the format shown in the figure; for example, they may be arranged vertically.

[0084] Specifically, first, the processor 13a selects an anomaly map corresponding to the verification image for each layer. In the example shown in Figure 9, for layers 4, 3, and 2, anomaly maps 311 and 312 corresponding to the selected verification images are selected and displayed side by side. The display control unit 13c then displays the composite anomaly maps 313 and 314 selected by the processor 13a in the anomaly map display area 203. The upper composite anomaly map 313 corresponds to the verification image 301 consisting of images of good products, and the lower composite anomaly map 314 corresponds to the verification image 302 consisting of images of defective products.

[0085] In the user interface screen 200 shown in Figure 9, a numerical display area 341 is provided where the degree of separation is displayed numerically for each level. This allows the user to easily understand the degree of separation for each level. In addition, the user interface screen 200 shown in Figure 9 also provides a histogram display area 342 where a cumulative histogram is displayed for each level.

[0086] In the anomaly severity maps 312 for layers 4, 3, and 2 shown in Figure 9, it can be seen that layer 2 has the highest degree of separation, including false positives. However, because it has false positives, its actual detection performance in operation is inferior to that of layers 4 and 3 (layers that do not have false positives). Since the processor 13a automatically selects the layer with the highest degree of separation, even if the actual detection performance is inferior to that of layers 4 and 3, it presents the anomaly severity map 312 for layer 2 to the user, as shown in Figure 8.

[0087] By transitioning from the user interface screen 200 shown in Figure 8 to the user interface screen 200 shown in Figure 9, the user can select the appropriate hierarchy while viewing the anomaly severity maps 311 and 312 in accordance with the actual detection request. In the example shown in Figure 9, the composite result (composite anomaly severity map) that appears to have the most stable detection performance is selected. If the composite result and the results of each hierarchy are the same, the composite result with higher stability may be selected provisionally.

[0088] The user selects the anomaly severity map. That is, the input unit 13d is configured to accept the user's selection of any anomaly severity map from among the multiple anomaly severity maps 311 to 314 displayed on the user interface screen 200. For example, when the input unit 13d detects an operation such as that of the mouse 52 and detects that the "composite result (composite anomaly severity maps 313, 314)" has been selected, the display control unit 13c displays a selection frame 340 surrounding the "composite result" on the user interface screen 200, as shown in Figure 10.

[0089] The display control unit 13c displays the degree of separation based on the abnormality map selected by the user in a cumulative histogram 320 in a separation degree display area 204, which is separate from the abnormality map display area 203 on the user interface screen 200. The cumulative histogram 320 on the separation degree display area 204 is provided with threshold adjustment bars 321 and 322 for adjusting the threshold. That is, the adjustment reception unit 13e shown in Figure 2 is the part that accepts user adjustments to the threshold, and superimposes the threshold adjustment bars 321 and 322, which can be operated left and right, onto the cumulative histogram 320. The user changes the threshold by moving one or both of the threshold adjustment bars 321 and 322 left or right with a mouse 52 or the like. This change in the threshold is accepted by the adjustment reception unit 13e.

[0090] The above describes the adjustment process in step SA9. Once the adjustment process is complete, the adjusted anomaly maps 311 and 312 are displayed in the anomaly map display area 203, as shown in Figure 11. In step SA10, if there is any discrepancy in the adjusted anomaly maps 311 and 312 shown in Figure 11, the result is determined to be YES, and the process proceeds to step SA1, where operations such as adding training images and retraining the model are performed. On the other hand, in step SA10, if there is no discrepancy in the adjusted anomaly maps 311 and 312 shown in Figure 11, the training process is terminated.

[0091] The following describes the case where, after the determination in step SA8, the system switches to debug mode, and the display control unit 13c displays the user interface screen 200 shown in Figure 12 on the display device 4. Focusing on the abnormality map 312 corresponding to the verification image 302 consisting of images of defective products in the abnormality map display area 203 shown in Figure 12, we see that in all layers, layers 4, 3, and 2, overdetection occurs not only in the defective area Wa, but also in the pattern and contour of the workpiece W.

[0092] Step SA8 is followed by Step SA9, where the adjustment process described above is performed (see Figure 13). However, since overdetection occurs at all levels, overdetection also occurs in the selected anomaly map 312 after adjustment (see Figure 14). Therefore, in Step SA10, it is determined that there is an anomaly, and the process proceeds to Step SA1, where operations such as adding training images and retraining the model are performed.

[0093] (Inspection process) Next, the operation of the visual inspection device 1 will be explained based on the flowchart shown in Figure 15. In step SC1 after startup, the processor 13a reads parameters stored in the memory device 19 and constructs a trained neural network and a trained model in the processor 13a and memory 13b. In step SC2, the workpiece to be inspected is captured by the imaging unit 3 to obtain a workpiece image. Then, the process proceeds to step SC3, where the workpiece image obtained in step SC2 is input into the convolutional neural network.

[0094] Next, in step SC4, feature maps containing the features of the work image are output from each layer of the convolutional neural network. Then, the feature maps output from each layer are applied to the model constructed in step SC1. This is the inference process.

[0095] Subsequently, in step SC5, the model outputs an anomaly severity map indicating the degree of anomaly for each layer. Then, in step SC6, an inspection process is executed to detect anomalies contained in the work image based on the output anomaly severity maps. Steps SC2 to SC6 can be executed each time the work changes. The processes shown in this flowchart correspond to the inspection steps executed by processor 13a.

[0096] (Effects of the embodiment) As explained above, during the training process, when a training image is input to a convolutional neural network, feature maps containing the characteristics of the training image are output from each layer of the convolutional neural network. Feature maps output from higher layers of the convolutional neural network have a higher level of abstraction and represent more widespread anomalies, while feature maps output from lower layers have a lower level of abstraction and represent more localized anomalies. Based on these feature maps with different characteristics, the modeling means generates a model.

[0097] Therefore, instead of pixel-based modeling, the feature map output from the convolutional neural network is modeled directly, eliminating detection errors due to general-purpose performance. In addition, it becomes possible to detect defects of various sizes at a single scale, resulting in improved anomaly detection performance across a wide range of sizes.

[0098] Furthermore, since processing at multiple scales is unnecessary and processing is performed at the smallest scale, the speed of training and inference processing is increased. In addition, it is robust to fluctuations, has few false positives, and has a high ability to detect trend anomalies in backgrounds with large variability, resulting in high robustness. Moreover, the increase in processing load due to color support is minimal, and has almost no impact on training and inference processing. Furthermore, image segmentation is unnecessary, and features within the receptive field can be aggregated using a single-pixel feature vector, thus further enhancing robustness.

[0099] The embodiments described above are merely illustrative in all respects and should not be interpreted restrictively. Furthermore, any modifications or changes that fall within the equivalent scope of the claims are all within the scope of the present invention.

[0100] For example, as shown in Figure 16, contrary to typical autoencoders, the number of channels in a layer may be compressed first and then expanded. The autoencoder is trained to match the input feature map output. In this example, to leverage the relationships between features, a 3x3 convolutional layer is used in each layer, and only compression and expansion in the channel direction are performed without spatial reduction. However, spatial reduction may be performed depending on the balance between performance and speed.

[0101] Figure 17 shows an example of operation, where the extracted feature map is input to a pre-trained autoencoder of the corresponding hierarchical level for inference processing. Specifically, the square of the norm of the difference between the corresponding local features at each location in the input and output feature maps is calculated, and the resulting value is used as the anomaly score for the corresponding location to generate an anomaly score map.

[0102] Figure 18 shows a case where processing is accelerated. Specifically, during inference, the trained CAE (shown by the box 500) is directly integrated with the trained neural network. This reduces the computational cost of the CAE and allows the parallel processing in the convolutional layers of the neural network to be reused. [Industrial applicability]

[0103] As described above, the present invention can be used when inspecting the appearance of a workpiece. [Explanation of symbols]

[0104] 1. Visual inspection device 4 Display device (display section) 13a processor 13b Memory 13c Display Control Unit 13d Input section 13e Adjustment Reception Department

Claims

1. An appearance inspection device comprising a processor and memory for constructing a convolutional neural network having a convolutional layer and a modeling means for generating a model for detecting anomalies based on a feature map output from the neural network, wherein the device uses the convolutional neural network and the modeling means to detect anomalies contained in an image to be inspected, The aforementioned processor, A learning process in which images of good products corresponding to good products are input to the convolutional neural network as training images, feature maps having the characteristics of the good product images are output from each layer of the convolutional neural network, and based on the feature maps output from each layer, multiple models for detecting anomalies are generated by the modeling means for each layer. The process involves inputting the image to be inspected into the convolutional neural network, outputting a feature map containing the characteristics of the image to be inspected from each layer of the convolutional neural network, applying the feature maps output from each layer to the model generated by the learning process, outputting an anomaly score map indicating the degree of anomaly for each layer, and performing an inspection process to detect anomalies contained in the image to be inspected based on the multiple anomaly score maps output. A storage unit for storing verification images for verifying the anomaly detection performance of the model generated by the modeling means, The system further comprises a display control unit for controlling the display unit, The processor outputs an abnormality map corresponding to the verification image for each layer using a plurality of models based on the verification image read from the storage unit, and the visual inspection device calculates the degree of separation between good product images and defective product images in each layer based on the plurality of output abnormality maps or a composite abnormality map obtained by combining the plurality of abnormality maps.

2. In the visual inspection apparatus according to claim 1, The modeling means is configured with an autoencoder and includes an encoding layer for compressing the feature map and a decoding layer for restoring the compressed feature map, in an appearance inspection device.

3. In the visual inspection apparatus according to claim 1 or 2, The aforementioned convolutional neural network is a visual inspection device in which the number of dimensions of the multidimensional vectors at the local locations of the feature map increases as one progresses from lower to higher layers.

4. In the visual inspection apparatus according to any one of claims 1 to 3, The display control unit generates a user interface screen capable of displaying the degree of separation at each level calculated by the processor and the degree of abnormality at each level, and displays the generated user interface screen on the display unit.

5. In the visual inspection apparatus according to any one of claims 1 to 4, The processor is configured to select the abnormality map or composite abnormality map output from the model with the highest calculated separation degree.

6. In the visual inspection apparatus according to any one of claims 1 to 5, An appearance inspection device further comprising an adjustment acceptance unit that accepts user adjustments to a threshold for determining whether a product is good or defective, compared with a score indicating the degree of abnormality of the aforementioned verification image.

7. In the visual inspection apparatus according to claim 6, The processor is configured to detect anomalies in the image to be inspected based on a plurality of anomaly severity maps output from each model, and to determine whether the verification image is a good product image or a defective product image based on the detected anomalies. The display control unit generates a cumulative histogram based on the frequency at which the verification image is determined to be a good image and the frequency at which the verification image is determined to be a defective image, and displays it on the display unit. The adjustment reception unit is configured to accept adjustments to the threshold value on the cumulative histogram displayed on the display unit.

8. In the visual inspection apparatus according to any one of claims 1 to 7, The processor generates a composite anomaly map by combining a plurality of anomaly maps, calculates the degree of separation of the verification image based on the generated composite anomaly map or the anomaly maps output from each layer of the model, and selects the composite anomaly map if the degree of separation calculated based on the composite anomaly map is equal to or greater than a predetermined value. The display control unit generates a user interface screen capable of displaying the composite anomaly degree map selected by the processor, and displays the generated user interface screen on the display unit.

9. In the visual inspection apparatus according to any one of claims 1 to 8, It further includes an input section that accepts user input on whether or not to start the learning adjustment mode, When the input unit receives notification of the start of the learning adjustment mode, the processor selects an anomaly score map or a composite anomaly score map corresponding to the verification images of each hierarchical level. The display control unit generates a user interface screen having a list display area capable of displaying anomaly maps corresponding to the verification images of each hierarchical level selected by the processor in a list format, and displays the generated user interface screen on the display unit.

10. In the visual inspection apparatus according to claim 9, The input unit is configured to accept the user's selection of any anomaly severity map from among the multiple anomaly severity maps displayed on the user interface screen. The display control unit is a visual inspection device that displays the degree of separation based on the abnormality map selected by the user as a cumulative histogram in a display area separate from the list display area on the user interface screen.

11. A visual inspection method for detecting anomalies contained in an image to be inspected, using a convolutional neural network having convolutional layers and a modeling means for generating a model for detecting anomalies based on a feature map output from the convolutional neural network, A learning step in which images of good products corresponding to good products are input to the convolutional neural network as training images, feature maps having the characteristics of the good product images are output from each layer of the convolutional neural network, and multiple models for detecting anomalies are generated by the modeling means based on the feature maps output from each layer, An inspection step in which the image to be inspected is input into the convolutional neural network, feature maps having the characteristics of the image to be inspected are output from each layer of the convolutional neural network, the feature maps output from each layer are applied to the model generated in the learning step, an anomaly score map indicating the degree of anomaly is output for each layer, and an anomaly contained in the image to be inspected is detected based on the multiple anomaly score maps output. The steps include storing a verification image in a storage unit for verifying the anomaly detection performance of the model generated by the modeling means, An appearance inspection method comprising the steps of: outputting an abnormality map corresponding to the verification image for each layer using a plurality of models based on the verification image read from the storage unit; and calculating the degree of separation between good product images and defective product images in the verification image at each layer based on the plurality of output abnormality maps or a composite abnormality map obtained by combining the plurality of abnormality maps.

Images