Information processing apparatus, information display apparatus, information processing method, information display method, and program

Abstract

To obtain a non-defective model for highly accurate abnormality detection even in a situation where the number of samples of non-defective images is insufficient.SOLUTION: The information processing apparatus 1 includes the feature quantity extractor 110 that extracts a feature quantity from a plurality of non-defective product images 50 of an inspection target, the division method determiner 120 that divides a non-defective product image into a plurality of small regions and groups the plurality of small regions into a plurality of groups based on the similarity of the distribution of the feature quantity of each small region, and the model constructor 140 that constructs a non-defective product model for each group using the feature quantity of the small region belonging to each group.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information display device, an information processing method, an information display method, and a program. [Background technology]

[0002] Conventionally, there has been known an inspection system that detects abnormalities in an inspection object based on an image of the inspection object. For example, Patent Document 1 discloses an inspection system that subdivides an image of a known inspection object to generate multiple partial images, and performs an inspection on each partial image using a machine learning model created for each partial image. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-131559 Summary of the Invention [Problem to be solved by the invention]

[0004] To improve the accuracy of an anomaly detection model, it is better to use a large number of samples of good product images to build the model. However, in reality, only a small number of good product image samples are often available, and building a model with a small number of samples can lead to overfitting, resulting in a problem of low accuracy in anomaly detection. In the system of Patent Document 1, a different machine learning model is created for each partial image, but further ingenuity is required to build a highly accurate model with a small number of good product image samples.

[0005] The present invention has been made in consideration of such problems, and one of its objectives is to make it possible to obtain a good product model for highly accurate anomaly detection even in a situation where the number of good product image samples is insufficient. [Means for solving the problem]

[0006] In order to solve the above-mentioned problems, the present invention employs the following configuration. An information processing device according to one aspect of the present invention includes a feature extraction unit that extracts features from a plurality of good product images of an inspection object; a division method determination unit that divides the good product images into a plurality of small regions and groups the small regions into a plurality of groups based on the similarity in the distribution of the features of each small region; and a model construction unit that constructs a good product model for each group using the features of the small regions belonging to each group.

[0007] With the above configuration, a non-defective product model can be constructed for each group having similar distributions of feature quantities, so that a non-defective product model with high accuracy in detecting anomalies can be constructed even in a situation where there are insufficient samples of non-defective product images.

[0008] The system may also be configured to include a feature extraction unit that extracts features from an inspection image of the inspection object, a region division unit that divides the inspection image into a plurality of regions corresponding to groups, and an abnormality calculation unit that calculates the degree of abnormality for each small region using a good product model of the corresponding group.

[0009] With the above configuration, it is possible to calculate the degree of abnormality for each small region using a non-defective product model that matches the characteristics of the region.

[0010] The image processing device may also be configured to include an abnormality degree integration unit that calculates the abnormality degree of the entire inspection image based on the abnormality degree calculated for each small region.

[0011] With the above configuration, the abnormality degrees calculated for each small region can be integrated to perform a highly accurate pass / fail judgment.

[0012] An information display device according to one aspect of the present invention includes an image display unit that displays an image of an object to be inspected, a mask display unit that displays a mask indicating boundaries that divide the image into multiple regions superimposed on the image, and a change operation receiving unit that receives operations to change the position of the boundaries, and constructs a good product model for each region divided by the boundaries using feature quantities of multiple small regions contained in each region.

[0013] With the above configuration, the user can check the appropriateness of the divided regions, and can change the division method to a more appropriate one as needed.

[0014] The apparatus may also be configured to include an inspection accuracy display unit that displays the inspection accuracy of the entire image when the degree of abnormality is calculated using the corresponding non-defective product model for each region divided by the boundaries.

[0015] With the above configuration, the user can check the inspection accuracy of the divided regions, and can change to a more appropriate division method based on the inspection accuracy.

[0016] The inspection accuracy display unit may be configured to display the inspection accuracy of the entire image corresponding to the boundary positions before and after the change operation so that the inspection accuracy can be compared.

[0017] With the above configuration, the user can check the division method while comparing the inspection accuracy results corresponding to the boundary positions before and after the change operation.

[0018] An information processing method according to one aspect of the present invention includes a feature extraction step of extracting features from a plurality of good product images of an inspection object; a division method determination step of dividing the good product images into a plurality of small regions and grouping the small regions into a plurality of groups based on the similarity in the distribution of the features of each small region; and a model construction step of constructing a good product model for each group using the features of the small regions belonging to each group.

[0019] With the above configuration, a non-defective product model can be constructed for each group having similar distributions of feature quantities, so that a non-defective product model with high accuracy in detecting anomalies can be constructed even in a situation where there are insufficient samples of non-defective product images.

[0020] An information display method according to one aspect of the present invention includes an image display step of displaying an image of an object to be inspected, a mask display step of displaying a mask indicating boundaries dividing the image into a plurality of regions superimposed on the image, a change operation receiving step of receiving an operation to change the position of the boundaries, and a step of constructing a good product model for each region divided by the boundaries using features of a plurality of small regions contained in each region.

[0021] With the above configuration, the user can check the appropriateness of the divided regions, and can change the division method to a more appropriate one as needed.

[0022] A program according to one aspect of the present invention causes a computer to execute a feature extraction step of extracting features from a plurality of good-quality product images of an inspection object; a division method determination step of dividing the good-quality product images into a plurality of small regions and grouping the small regions into a plurality of groups based on the similarity in the distribution of the features of each small region; and a model construction step of constructing a good-quality product model for each group using the features of the small regions belonging to each group.

[0023] With the above configuration, a non-defective product model can be constructed for each group having similar distributions of feature quantities, so that a non-defective product model with high accuracy in detecting anomalies can be constructed even in a situation where there are insufficient samples of non-defective product images.

[0024] A program according to one aspect of the present invention causes a computer to execute the following steps: an image display step of displaying an image of an object to be inspected; a mask display step of displaying a mask indicating boundaries that divide the image into multiple regions superimposed on the image; a change operation reception step of receiving an operation to change the position of the boundaries; and a step of constructing a good product model for each region divided by the boundaries using features of multiple small regions contained in each region.

[0025] With the above configuration, the user can check the appropriateness of the divided regions, and can change the division method to a more appropriate one as needed. [Effects of the Invention]

[0026] According to the present invention, even in a situation where the number of samples of non-defective product images is insufficient, a non-defective product model for highly accurate anomaly detection can be obtained. [Brief explanation of the drawings]

[0027] [Figure 1] 1A and 1B are diagrams showing examples of a non-defective product image and an inspection image according to an embodiment of the present invention. [Figure 2] 1 is a diagram showing an example of the hardware configuration of an information processing device (information display device) and an information processing terminal according to an embodiment of the present invention. [Figure 3] 1 is a diagram showing an example of the functional configuration of an information processing device (information display device) according to an embodiment of the present invention. [Figure 4] FIG. 4 is a diagram showing an example of a screen displayed on a display according to an embodiment of the present invention. [Figure 5] FIG. 10 is a diagram showing another example of a screen displayed on the display according to the embodiment of the present invention. [Figure 6] 1 is a flowchart showing a procedure for constructing a non-defective product model according to an embodiment of the present invention. [Figure 7] 10 is a flowchart showing a procedure for calculating the degree of abnormality of an inspection image according to an embodiment of the present invention. [Figure 8] 10 is a flowchart illustrating a procedure for a user to check and correct a divided area according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0028] An embodiment according to one aspect of the present invention (hereinafter also referred to as "the present embodiment") will be described below with reference to the drawings. However, the embodiment described below is merely an example of the present invention in all respects. Needless to say, various improvements and modifications can be made without departing from the scope of the present invention. In other words, when implementing the present invention, specific configurations according to the embodiment may be appropriately adopted. Note that, although data appearing in this embodiment is described in natural language, more specifically, it may be specified in any of computer-recognizable pseudo-language, commands, parameters, or machine language, but is not limited to these.

[0029] <Summary> The present invention is applicable to, for example, an apparatus that uses a machine learning model to perform visual inspection of images of inspected workpieces, such as industrial products, and detect anomalies. The apparatus according to the present invention has a function for constructing a non-defective product model for calculating the degree of anomaly of an inspection image 60, and a function for calculating the degree of anomaly of the inspection image 60 using the constructed non-defective product model. The non-defective product model is constructed using features extracted from a non-defective product image 50 of the inspected workpiece. Existing image processing techniques can be used to extract features from the non-defective product image 50. Examples include SIFT (Scale-Invariant Feature Transform), HOG (Histograms of Oriented Gradients), SURF (Speeded Up Robust Features), and LBP (Local Binary Pattern). Other examples include trained convolutional neural networks (CNNs) and Haar.

[0030] Figure 1 shows an example of a non-defective image (A) and an inspection image (B) of a workpiece to be inspected. It is desirable that the non-defective image 50 and the inspection image 60 are as similar as possible in terms of image size, alignment, lighting, and other conditions during image capture. Alternatively, these conditions can be corrected after capture to make them identical. As shown in Figure 1, both images contain background regions such as small regions 51 and 52 and workpiece edge regions (the boundary between the workpiece and the background) such as small regions 53, 54, and 55, due to factors such as alignment during image capture. The features extracted from each image are position-dependent, and feature values ​​at corresponding positions tend to be similar even in different non-defective images. Furthermore, within a single image, there are small regions with similar feature values. For example, in Figure 1, small regions 51 and 52 are located in the background region, and the feature values ​​of each small region are similar. Similarly, small regions 54 and 55 are both located in the workpiece edge region, and the feature values ​​of each small region are similar. On the other hand, although the small region 53 is located in the workpiece edge region, the similarity between the feature amount of the small region 53 and the feature amounts of the small regions 51, 52, 54, and 55 is low.

[0031] In this invention, a good-quality product image 50 is divided into multiple small regions, and small regions with similar features, such as small regions 51 and 52 and small regions 54 and 55, are grouped together. A good-quality product model is constructed for each group. For example, if there are 10 samples of good-quality product images, 10 samples of features are extracted from each small region. Small regions with similar feature distributions are grouped together, and the features of the small regions belonging to the same group are used as the features of the entire group to construct a good-quality product model for each group. In other words, if a group contains five small regions, five times as many features can be obtained, or 50 samples. In this way, even if the original number of samples is small, by grouping similar small regions, a good-quality product model can be constructed using a pseudo-large number of samples, thereby improving the accuracy of the good-quality product model. Note that the distribution of features may be expressed as the range of a sample set of features plotted in a feature space, or may be expressed as a probability distribution by calculating statistics of multiple features. In other words, the distribution of features may be either a discrete distribution or a continuous distribution.

[0032] When calculating the degree of abnormality of an inspection image using the constructed non-defective product model, the inspection image is divided into small regions, and the degree of abnormality is calculated using the non-defective product model of the group to which each small region belongs.

[0033] <Configuration> FIG. 2 is a diagram showing an example of the hardware configuration of an information processing device (information display device) 1 according to this embodiment. The information processing device 1 is a computer including a processor 11, a main memory 12, an input / output interface 13, a communication interface 14, and a storage device 15. The storage device 15 is a computer-readable recording medium such as a semiconductor memory (which may be, but is not limited to, a volatile memory or a non-volatile memory) or a disk medium (which may be, but is not limited to, a magnetic recording medium or a magneto-optical recording medium). The storage device 15 stores a program to be executed by the processor 11. The program is loaded from the storage device 15 into the main memory 12 and interpreted and executed by the processor 11. The storage device 15 also includes a database 16. The database 16 may be implemented in an external storage device. The database 16 stores, for example, constructed non-defective product models.

[0034] The information processing device 1 is connected to an information processing terminal 2 via a communication network. There may be multiple information processing terminals 2. The information processing device 1 and the information processing terminal 2 may be implemented as the same device. The information processing device 1 is, for example, a server device, but is not limited to a server device and may also be a computer such as a desktop, laptop, or tablet. The information processing terminal 2 may also be a computer such as a smartphone, mobile phone, desktop, laptop, or tablet. The information processing terminal 2 has a control unit 24 including a CPU and memory, a communication unit 25 for connecting to the network N, an input unit 21 such as a touch panel or keyboard for accepting operations from a user, a storage unit 26 in which various programs and data required for executing processing in the control unit 24 are stored, and a display 23 for displaying a screen.

[0035] <Functional configuration> Fig. 3 is a diagram illustrating an example of the functional configuration of the information processing device 1. As illustrated in Fig. 3, the information processing device 1 includes an image input unit 100, a feature extraction unit 110, a division method determination unit 120, an area division unit 130, a model construction unit 140, an anomaly level calculation unit 150, an anomaly level integration unit 151, an image display unit 170, a mask display unit 171, a change operation acceptance unit 180, and an inspection accuracy display unit 181, which are functional modules executed by the processor 11. In addition, the division method determination unit 120 includes a tile extraction unit 121, a similarity measurement unit 122, and a grouping unit 123.

[0036] The image input unit 100 has a function of accepting input of a non-defective product image 50 used to construct a non-defective product model and an inspection image 60 for calculating an abnormality degree. The non-defective product image 50 and the inspection image 60 may be images captured by a known camera.

[0037] The feature extraction unit 110 extracts features from the input non-defective product image 50 or inspection image 60. Any known method can be used to extract the features. For example, the feature extraction unit 110 may extract the features using a feature extractor that retains spatial information, such as CNN or Haar.

[0038] The division method determination unit 120 divides the non-defective image 50 into a plurality of small regions (hereinafter also referred to as tiles), and groups the plurality of tiles into a plurality of groups based on the similarity of the distribution of the feature amounts of each tile. The division method determination unit 120 includes a tile extraction unit 121, a similarity measurement unit 122, and a grouping unit 123.

[0039] The tile extraction unit 121 divides the non-defective image 50 into a plurality of tiles with a predetermined tile size and stride, and extracts the feature amount of each tile. The tile size and stride may be set to arbitrary values ​​by the user.

[0040] The smaller each tile, the more likely it is that grouping will occur. On the other hand, if there are spots on the workpiece surface, a moderate size may make it easier to distinguish them from noise. Therefore, the tile size and stride settings can be changed to appropriate sizes depending on the properties of the workpiece being inspected.

[0041] The similarity measurement unit 122 calculates the similarity of the distribution of feature values ​​for each tile. The calculation of similarity may be performed using a method that uses statistical similarity, or an approximation method using clustering. For example, the similarity between all tiles may be calculated using statistical similarity such as KL distance, PE distance, optimal transport, or independence test.

[0042] Based on the calculated similarity, the grouping unit 123 classifies tiles with a certain level of similarity or higher into the same group. Alternatively, an approximation method using clustering may be used to treat multiple feature points of each tile as one feature vector, and to classify tiles with a certain level of similarity or higher into the same group by clustering.

[0043] For example, for vectors of the same size obtained from each tile, an approximation method using clustering such as K-means can be used to treat multiple feature points of each tile as a single feature vector, and clustering can be used to classify those with a certain level of similarity or higher into the same group.

[0044] The similarity measurement unit 122 and the grouping unit 123 may be integrated into one clustering processing unit.

[0045] The region dividing unit 130 divides the non-defective image 50 based on the division method determined by the division method determining unit 120 .

[0046] The model construction unit 140 constructs a good product model based on the feature quantities of the good product images 50. The model construction unit 140 constructs a good product model for a group by using the feature quantities for the good product images extracted for each small region belonging to the same group as the feature quantities of the group. For example, if there are n good product samples and m small regions are grouped together, a good product model is constructed for each small region using n good product samples. However, a good product model can be constructed for each group using n × m good product samples, i.e., m times the number of good product samples per small region. This enables higher performance than when a good product model is constructed for each small region, even when the number of good product samples is insufficient. Furthermore, by grouping small regions with similar feature distributions together, higher performance than conventional methods can be achieved even when the distribution of good product features is position-dependent.

[0047] The good product model is a model constructed based on the distribution of the feature quantities of good products, and is used, for example, to estimate the degree of abnormality. The good product model is expressed, for example, by statistics or a sample set of the feature quantities of good products. The model construction unit 140 may construct the good product model using, for example, a neural network.

[0048] Furthermore, by increasing the feature amount per non-defective product model while taking into consideration the position dependency of the feature amount distribution of the non-defective product image 50, the non-defective product distribution expressed by the model can approach the true non-defective product distribution, leading to accurate pass / fail judgment.

[0049] The image input unit 100 receives input of an inspection image 60. The inspection image 60 is an image captured by a known camera.

[0050] The feature extraction unit 110 extracts features from the inspection image 60 to reduce redundancy in the image data. Any known method can be used to extract the features. For example, the feature extraction unit 110 extracts features from the input image using a feature extractor that retains spatial information, such as CNN or Haar.

[0051] As one example, features established by existing image processing technologies such as Haar-like features and Hog may be extracted, or features may be extracted using trained deep learning, such as CNN, as a feature extractor. In this case, a model trained on general object recognition images or a model trained on defect images may be used.

[0052] The region dividing unit 130 divides the inspection image 60 based on the division method determined by the division method determining unit 120 .

[0053] The anomaly degree calculation unit 150 constructs a non-defective product model in the learning stage using the feature amounts of the non-defective product image 50, and uses the non-defective product model to calculate the degree of anomaly for the feature amounts of the inspection image 60. The anomaly degree calculation unit 150 calculates the degree of anomaly for the feature amounts of the target image based on the distance or similarity to the non-defective product model constructed in the learning stage.

[0054] The anomaly degree calculation unit 150 calculates the degree of anomaly by extracting feature values ​​from each small region of the inspection image 60 and calculating the distance from the non-defective product distribution. The smaller the distance, the more similar the feature values ​​are to the non-defective product image, and the more likely the inspected workpiece is a non-defective product. The degree of anomaly may also be calculated using statistics (mean and covariance) of the non-defective product model. In this case, the feature values ​​of the non-defective product image may be assumed to be distributed according to a multivariate normal distribution, and the inverse of the probability density for the feature values ​​may be used as the degree of anomaly.

[0055] If the degree of abnormality calculated by the abnormality degree calculation unit 150 is equal to or greater than a predetermined value, the abnormality degree calculation unit 150 determines the test result as pass, and if the degree of abnormality is less than the predetermined value, the abnormality degree calculation unit 150 determines the test result as fail.

[0056] The abnormality degree calculation unit 150 may calculate the inspection performance. The inspection results determined based on the degree of abnormality calculated by the abnormality degree calculation unit 150 are tallied, and the inspection performance can be calculated by reflecting results such as a result that was determined to be pass but actually turned out to be fail.

[0057] The inspection performance may be calculated using, for example, a receiver operating characteristic (ROC) curve. More specifically, the ROC may be calculated using a plurality of images of non-defective and defective products prepared in advance, and the area under the curve (AUC) may be calculated and used as an index of inspection performance. Alternatively, the false positive rate and false negative rate may be calculated and used as an index of inspection performance.

[0058] For each divided region, the anomaly degree calculation unit 150 calculates the degree of anomaly for the feature quantities of the inspection image 60 using a good product model constructed in the learning stage using the feature quantities of the good product image 50. For example, the degree of anomaly may be calculated based on the distance between each feature quantity in each tile and the nearest point of the set of stored feature quantities of good products corresponding to each feature quantity in each tile during inference, and the degree of anomaly for each tile may be integrated to calculate the degree of anomaly for the entire image.

[0059] The abnormality degree integrating unit 151 may integrate the calculated abnormality degrees for each divided small region and output the integrated abnormality degree. The abnormality degree integrating unit 151 may also perform processing to calculate the maximum abnormality degree, the average abnormality degree, etc., and output the integrated abnormality degree.

[0060] For example, an improved method with high noise resistance can be implemented by clustering small regions based on the degree of anomaly, extracting clusters with areas larger than a predetermined threshold, and calculating the average degree of anomaly for each small region included in the cluster to obtain an integrated degree of anomaly. This method not only calculates the degree of anomaly, but also identifies the location of an anomaly if the inspection result is a failure. Additionally, by displaying the clusters on the display 23 so that they overlap with the inspection image 60, the user can identify the location of an anomaly in the workpiece being inspected, thereby assisting in analyzing the cause of the anomaly.

[0061] The abnormality level integrating unit 151 may integrate the test results for each divided small region and output the integrated test result. For example, the abnormality level integrating unit 151 may determine the integrated test result as failing when there are a predetermined number or more (for example, one) of failures among the test results for each divided small region. If there are no failures, the abnormality level integrating unit 151 may determine the integrated test result as passing.

[0062] The small regions may be pixels of an image, or may be square or rectangular regions. The anomaly degree calculation unit 150 calculates the anomaly degree for each pixel, for example, based on the distance to the non-defective product feature distribution, but the anomaly degree may be calculated using other methods, or multiple calculation methods may be employed. In this case, the anomaly degree integration unit 151 may integrate the results of the multiple calculation methods and output the anomaly degree.

[0063] The image display unit 170 displays the image showing the divided areas on the screen, superimposed on any of the non-defective product images 50 or inspection images 60. The area division status and inspection results can be confirmed from the display screen controlled by the image display unit 170.

[0064] The image display unit 170 can switch between the non-defective image 50 and the inspection image 60, for example, in response to an instruction from the user.

[0065] The image display unit 170 can display, for example, the following items as checkable items on the screen: Multiple region division patterns - Abnormality level and judgment for each image Inspection performance

[0066] The image display unit 170 may display an editing screen, a display selection screen, etc., which will be described later. For example, the image display unit 170 may display a plurality of area division patterns, allow the user to select a preferred pattern, and receive the result from the change operation receiving unit 180, which will be described later.

[0067] The mask display unit 171 receives the change in the division method instructed by the user via the change operation receiving unit 180, which will be described later, and displays a mask M corresponding to the changed division method on the display 23. As a result, the mask M indicating the boundaries dividing the image into a plurality of divided regions can be superimposed on the image displayed by the image display unit 170.

[0068] The mask display unit 171 may receive the division method of the area received by the division method determination unit 120 and display a mask M corresponding to the division method on the display 23, or may display a mask M corresponding to an area divided by any method.

[0069] The mask M displayed by the mask display unit 171 may be filled with different colors so that the divided areas of the same group are the same color. Note that a predetermined transparency may be provided so that the underlying image can be seen. In addition to color coding, the mask M may be filled with hatching.

[0070] The change operation receiving unit 180 receives an instruction to change the divided areas from the user. For example, the user issues an instruction to change the area division method on an editing screen from a display screen controlled by the image display unit 170, and the change operation receiving unit 180 outputs the received instruction as division method data to the area dividing unit 130.

[0071] The change operation receiving unit 180 may receive an instruction to reevaluate the changed region division. For example, when the user operates a reevaluation button displayed on the image display unit 170 to instruct calculation of the degree of abnormality, the inspection accuracy display unit 181 (described later) can display, for each changed region division, the inspection accuracy of the entire image when the degree of abnormality is calculated using the corresponding non-defective product model.

[0072] As a result, the region division state can be visualized on the image, allowing the user to check and change the division state.By changing the region division method, the user can interpret and intervene in the degree of abnormality and judgment results.

[0073] For example, a user can view the division status overlaid on the image and compare it with the inspection performance to see how the division of the region has affected the model performance.

[0074] The user can also determine whether the inspection area is correctly divided into subareas to which different inspection methods should be applied.

[0075] The inspection accuracy display unit 181 may display on the screen the inspection accuracy of the entire image when the degree of abnormality is calculated using the corresponding non-defective product model for each region divided by the boundaries.

[0076] The inspection accuracy display unit 181 displays the inspection accuracy of the entire image when the degree of abnormality is calculated using the corresponding non-defective product model for each area divided by the boundary, allowing the user to easily try and error by checking the relationship between the division method and the judgment result.

[0077] As described above, the user can check the inspection accuracy of the divided regions, and can change to a more appropriate division method based on the inspection accuracy.

[0078] Furthermore, the inspection accuracy display unit 181 may display the inspection accuracy of the entire image corresponding to the boundary positions before and after the change operation in a manner that allows comparison.

[0079] For a plurality of images divided by different division methods, the inspection accuracy of the entire plurality of images can be calculated, and the calculation results can be displayed on the screen for each image by the inspection accuracy display unit 181.

[0080] As described above, the user can check the division method while comparing the inspection accuracy results corresponding to the boundary positions before and after the change operation.

[0081] <Screen example> FIG. 4 is a diagram illustrating an example of a screen displayed on the display 23. Using FIG. 4, the operation of the user to check and correct the divided areas will be specifically described. First, as shown in FIG. 4(A), the display 23 displays an image (left side of the screen) in which a mask M (defining areas R1, R2, and R3) is superimposed on the work P, and an image (right side of the screen) in which an abnormality map is superimposed on the work P. The abnormality map is, for example, a mask M in which each small area is filled with a color that increases in brightness as the degree of abnormality increases. Below the image on the right, the degree of abnormality calculated based on the divided areas shown on the left, the judgment result (judgment based on the degree of abnormality), and the inspection performance are displayed. Operating the "Edit Area" button displayed at the top left of the screen transitions to an editing screen such as that shown in FIG. 4(B).

[0082] On the edit screen, the range of each of the regions R1 to R3 can be changed using a mouse, etc. When the region confirmation button on the top right of the screen is operated, the screen transitions to the screen shown in Figure 4(C).

[0083] On the screen in Figure 4(C), the mask M after the area change is displayed on the left. As shown in Figure 4(C), in the changed mask M, part of the original area R2 has been changed to area R1. Clicking the "Reevaluate" button at the top right of the screen will transition to the screen shown in Figure 4(D).

[0084] The screen in FIG. 4(D) displays the recalculated abnormality level and inspection performance corresponding to the changed division area.

[0085] FIG. 5 is a diagram showing another example of a screen displayed on the display 23. In the example of FIG. 5, the user can select a division method from a plurality of options. As shown in FIG. 5, an image overlaid with a mask M, each of which has a different division method, is displayed on the left side of the screen, and the degree of abnormality and inspection performance calculated for each division method are displayed on the right side. The user can select a division method from the options. Note that the images on the right side of FIGS. 4(A), (C), (D) and FIG. 5 are not limited to the mode in which an anomaly map is overlaid, and the work P may be displayed without an anomaly map overlaid, or an abnormality may be indicated in another display mode.

[0086] <Operation> Next, the operation of the information processing device 1 according to this embodiment will be described.

[0087] First, the procedure for constructing a non-defective product model will be described with reference to the flowchart in FIG.

[0088] First, in step S101, the image input unit 100 receives input of a non-defective product image 50 of a workpiece to be inspected.

[0089] Next, in step S102, the feature extraction unit 110 extracts feature amounts from the input non-defective product image 50.

[0090] Next, in step S103, the tile extraction unit 121 divides the non-defective image 50 into a plurality of tiles with a predetermined tile size and stride, and extracts the feature amount of each tile.

[0091] Next, in step S104, the similarity measurement unit 122 calculates the similarity between each tile based on the distribution of the feature amount of each tile.

[0092] Next, in step S105, the grouping unit 123 groups the tiles into a plurality of groups based on the calculated similarities between the tiles.

[0093] Next, in step S106, the model construction unit 140 constructs a non-defective model for each group using the feature amounts of the tiles belonging to each group.

[0094] Next, a procedure for calculating the degree of abnormality of the inspection image 60 using the non-defective product model will be described with reference to the flowchart of FIG.

[0095] First, in step S201, the image input unit 100 receives input of an inspection image 60 of a workpiece to be inspected.

[0096] Next, in step S202, the feature extraction unit 110 extracts features from the input inspection image 60.

[0097] Next, in step S203, the region dividing unit 130 divides the inspection image 60 into a plurality of regions corresponding to the groups generated by the grouping unit 123.

[0098] Next, in step S204, the abnormality degree calculation unit 150 calculates the abnormality degree for each tile in the inspection image 60 using the non-defective product model of the corresponding group constructed by the model construction unit 140.

[0099] Next, in step S205, based on the degree of abnormality calculated for each divided region, the abnormality degree integrating unit 151 calculates the overall degree of abnormality of the inspection image 60. For example, the abnormality degree integrating unit 151 calculates the average value of the degrees of abnormality calculated by each abnormality degree calculating unit 150 to calculate the overall degree of abnormality of the inspection image 60.

[0100] Next, the procedure for the user to check and correct the divided areas will be described with reference to the flowchart in FIG.

[0101] First, in step S301, the image display unit 170 displays an image of the workpiece to be inspected on the display 23 of the information processing terminal 2. The image to be displayed may be a randomly selected non-defective product image 50 or an inspection image 60.

[0102] Next, in step S302, the mask display unit 171 displays a mask M indicating the boundaries dividing the image into a plurality of divided regions, superimposed on the image displayed in step S301. The displayed mask M may be a mask M indicating the boundaries divided into a plurality of divided regions using a method determined by the division method determination unit 120, or may be a mask M indicating the boundaries divided into a plurality of divided regions using any division method designated by the user. The mask M may be displayed in a different color for each divided region.

[0103] As shown in Figure 4, a mask M is displayed overlaid on the workpiece P. Mask M indicates the boundaries of three divided regions R1, R2, and R3. Region R2 is divided into two, with region R3 in between.

[0104] Next, in step S303, the inspection accuracy display unit 181 calculates and displays the degree of abnormality and inspection accuracy of the entire image when the degree of abnormality is calculated using the corresponding non-defective model for each divided area divided by the boundaries of the mask M.

[0105] The non-defective model used here is a non-defective model for each divided area constructed by the model construction unit 140. That is, it is a model constructed using the feature amounts of tiles belonging to each divided area. Note that the model construction unit 140 may construct one non-defective model for a divided area by using the feature amounts for the non-defective image extracted for each tile belonging to the same divided area as the feature amounts for the entire divided area.

[0106] Next, in step S304, the change operation receiving unit 180 determines whether or not an instruction to change the divided areas has been received from the user. For example, when the user presses the "Edit Area" button displayed on the screen, the change operation receiving unit 180 determines that an instruction to change the divided areas has been received from the user.

[0107] If the change operation receiving unit 180 determines that the user has pressed the "Edit Area" button (step S304: YES), the process proceeds to step S305. If the change operation receiving unit 180 determines that the user has not pressed the "Edit Area" button (step S304: NO), the process remains at step S304.

[0108] Next, in step S305, the change operation receiving unit 180 acquires operation information instructed by a user's operation on the screen and changes the division method. The user may instruct a change in the division method by operating the mouse on the editing screen, etc.

[0109] Next, in step S306, the mask display unit 171 receives the change in the division method instructed by the user, and displays a mask M corresponding to the changed division method on the display 23. Furthermore, the inspection accuracy display unit 181 displays on the screen the degree of abnormality and the inspection accuracy calculated corresponding to the changed division area. Note that the degree of abnormality and the inspection accuracy may be recalculated by the user operating a re-evaluation button displayed on the display 23. The inspection accuracy display unit 181 may display the degree of abnormality and the inspection accuracy before and after the operation to change the division area so that they can be compared.

[0110] Next, in step S307, the change operation receiving unit 180 determines whether or not an instruction to end the change of the divided area has been received from the user. For example, the change operation receiving unit 180 determines whether or not an instruction to end the change of the divided area has been received from the user when the user operates the "Confirm Area" button or the "Reevaluate" button displayed on the screen.

[0111] If it is determined that the user has pressed the "Confirm Area" button, the change operation receiving unit 180 determines that it has received an instruction from the user to end the change of the divided area (step S307: YES), the divided area is confirmed, and the processing ends.

[0112] If it is determined that the user has pressed the "reevaluate" button, the change operation receiving unit 180 determines that it has not received an instruction from the user to end the change of the divided area (step S307: NO), and proceeds to step S305, where the change operation receiving unit 180 performs the above-mentioned processing, such as acquiring operation information instructed by the user again operating on the screen.

[0113] As described above, according to the present invention, regions having a distribution of feature quantities are grouped, and a good product model is constructed for each group. In this case, the feature quantities of small regions belonging to the same group are used as the feature quantities of that group, so that it is possible to substantially increase the number of good product samples, and to construct a good product model with high accuracy.

[0114] Furthermore, when determining whether an inspection image is good or bad, the inspection image is divided into regions corresponding to each group, and the degree of abnormality is calculated using the good product model of the corresponding group for each region, thereby enabling accurate determination of whether the image is good or bad.

[0115] Furthermore, a mask M indicating the region division method is superimposed on the image of the inspection image 60 on the display 23, and the user can change the division method on the screen, so that the user can change to a more appropriate division method as needed. At this time, the abnormality level values ​​and inspection accuracy calculated using the division methods before and after the change can be displayed to assist the user in changing the divided regions.

[0116] The present embodiment is intended to facilitate understanding of the present invention and is not intended to limit the present invention. The present invention can be modified or improved without departing from the spirit thereof, and the present invention also includes equivalents thereof.

[0117] Furthermore, in the present invention, the term "unit" does not simply mean a physical means, but also includes cases where the functions of the "unit" are realized by software. Furthermore, the functions of one "unit" or device may be realized by two or more physical means, devices, or software, or the functions of two or more "units" or devices may be realized by one physical means, device, or software. Note that some or all of the above embodiments may also be described as in the following supplementary notes, but are not limited to the following.

[0118] (Appendix 1) a feature extraction unit that extracts feature amounts from a plurality of non-defective product images of an inspection object; a division method determination unit that divides the non-defective product image into a plurality of small regions and groups the plurality of small regions based on the similarity of distribution of feature amounts of each small region; a model construction unit that constructs a non-defective product model for each group using the feature values ​​of the small regions that belong to each group; Information processing device. (Appendix 2) a feature extraction unit that extracts feature amounts from an inspection image of the inspection object; an area dividing unit that divides the inspection image into a plurality of areas corresponding to the groups; an abnormality degree calculation unit that calculates an abnormality degree for each small region using the non-defective product model of the corresponding group, 10. The information processing device according to claim 1. (Appendix 3) an abnormality degree integration unit that calculates the abnormality degree of the entire inspection image based on the abnormality degrees calculated for each small region; 3. The information processing device according to claim 2. (Appendix 4) an image display unit that displays an image of the object to be inspected; a mask display unit that displays a mask indicating boundaries that divide the image into a plurality of regions, superimposed on the image; a change operation receiving unit that receives an operation to change the position of the boundary, constructing a non-defective product model for each region divided by the boundaries using feature amounts of a plurality of small regions included in each region; Information display device. (Appendix 5) an inspection accuracy display unit that displays, for each region divided by the boundary, the inspection accuracy of the entire image when the degree of abnormality is calculated using the corresponding non-defective model; 5. An information display device according to claim 4. (Appendix 6) The inspection accuracy display unit and displaying the inspection accuracy of the entire image corresponding to the boundary positions before and after the change operation in a manner that allows comparison. 6. An information display device according to claim 5. (Appendix 7) a feature extraction step of extracting feature amounts from a plurality of non-defective product images of an inspection object; a division method determination step of dividing the non-defective product image into a plurality of small regions and grouping the plurality of small regions into a plurality of groups based on the similarity of distribution of feature amounts of each small region; and a model construction step of constructing a non-defective product model for each group using the feature amounts of the small regions belonging to each group. An information processing method executed by an information processing device. (Appendix 8) an image display step of displaying an image of the object to be inspected; a mask display step of superimposing a mask indicating boundaries dividing the image into a plurality of regions on the image and displaying the mask; a change operation receiving step of receiving a change operation for the position of the boundary; and constructing a non-defective product model for each of the regions divided by the boundaries using feature amounts of a plurality of small regions included in each of the regions. An information display method executed by an information display device. (Appendix 9) On the computer, a feature extraction step of extracting feature amounts from a plurality of non-defective product images of an inspection object; a division method determination step of dividing the non-defective product image into a plurality of small regions and grouping the plurality of small regions into a plurality of groups based on the similarity of distribution of feature amounts of each small region; a model construction step of constructing a non-defective product model for each group using the feature values ​​of the small regions belonging to each group; A program to execute. (Appendix 10) On the computer, an image display step of displaying an image of the object to be inspected; a mask display step of superimposing a mask indicating boundaries dividing the image into a plurality of regions on the image and displaying the mask; a change operation receiving step of receiving a change operation for the position of the boundary; constructing a non-defective product model for each region divided by the boundaries using feature amounts of a plurality of small regions included in each region; A program to execute. [Explanation of symbols]

[0119] 1...information processing device (information display device), 2...information processing terminal, 11...processor, 12...main memory, 13...input / output interface, 14...communication interface, 15...storage device, 16...database, 21...input unit, 23...display, 24...control unit, 25...communication unit, 26...storage unit, 50...good product image, 60...inspection image, 100...image input unit, 110...feature extraction unit, 120...division method determination unit, 121...tile extraction unit, 122...similarity measurement unit, 123...grouping unit, 130...area division unit, 140...model construction unit, 150...anomaly calculation unit, 151...anomaly integration unit, 160...output unit, 170...image display unit, 171...mask display unit, 180...change operation reception unit, 181...inspection accuracy display unit, M...mask, P...work, R1...area, R2...area, R3...area

Claims

1. a feature extraction unit that extracts feature amounts from a plurality of non-defective product images of an inspection object; a division method determination unit that divides the non-defective product image into a plurality of small regions and groups the plurality of small regions based on the similarity of distribution of feature amounts of each small region; a model construction unit that constructs a non-defective product model for each group using the feature values ​​of the small regions that belong to each group; Information processing device.

2. a feature extraction unit that extracts feature amounts from an inspection image of the inspection object; an area dividing unit that divides the inspection image into a plurality of areas corresponding to the groups; an abnormality degree calculation unit that calculates an abnormality degree for each small region using the non-defective product model of the corresponding group, The information processing device according to claim 1 .

3. an abnormality degree integration unit that calculates the abnormality degree of the entire inspection image based on the abnormality degrees calculated for each small region; The information processing device according to claim 2 .

4. an image display unit that displays an image of the object to be inspected; a mask display unit that displays a mask indicating boundaries that divide the image into a plurality of regions, superimposed on the image; a change operation receiving unit that receives an operation to change the position of the boundary, constructing a non-defective product model for each region divided by the boundaries using feature amounts of a plurality of small regions included in each region; Information display device.

5. an inspection accuracy display unit that displays, for each region divided by the boundary, the inspection accuracy of the entire image when the degree of abnormality is calculated using the corresponding non-defective model; The information display device according to claim 4.

6. The inspection accuracy display unit and displaying the inspection accuracy of the entire image corresponding to the boundary positions before and after the change operation in a manner that allows comparison. The information display device according to claim 5 .

7. a feature extraction step of extracting feature amounts from a plurality of non-defective product images of an inspection object; a division method determination step of dividing the non-defective product image into a plurality of small regions and grouping the plurality of small regions into a plurality of groups based on the similarity of distribution of feature amounts of each small region; and a model construction step of constructing a non-defective product model for each group using the feature amounts of the small regions belonging to each group. An information processing method executed by an information processing device.

8. an image display step of displaying an image of the object to be inspected; a mask display step of superimposing a mask indicating boundaries dividing the image into a plurality of regions on the image and displaying the mask; a change operation receiving step of receiving a change operation for the position of the boundary; and constructing a non-defective product model for each of the regions divided by the boundaries using feature amounts of a plurality of small regions included in each of the regions. An information display method executed by an information display device.

9. On the computer, a feature extraction step of extracting feature amounts from a plurality of non-defective product images of an inspection object; a division method determination step of dividing the non-defective product image into a plurality of small regions and grouping the plurality of small regions into a plurality of groups based on the similarity of distribution of feature amounts of each small region; a model construction step of constructing a non-defective product model for each group using the feature values ​​of the small regions belonging to each group; A program to execute.

10. On the computer, an image display step of displaying an image of the object to be inspected; a mask display step of superimposing a mask indicating boundaries dividing the image into a plurality of regions on the image and displaying the mask; a change operation receiving step of receiving a change operation for the position of the boundary; constructing a non-defective product model for each region divided by the boundaries using feature amounts of a plurality of small regions included in each region; A program to execute.

Images