Defect analysis device, defect analysis method, and program

Abstract

The present invention analyzes the classification of a defect on the basis of an image having captured the defect. This defect analysis device comprises: a model storage unit for storing a feature extraction model that is generated by performing learning on the basis of learning data obtained by adding information indicating classifications of flaws to defect images having captured therein flaws generated on the surface of an object, and that extracts, from an input image, an image feature quantity with which the distance between similar images becomes small; a representative point determination unit for determining representative points for the respective classifications on the basis of the image feature quantity extracted from the defect image by using the feature extraction model; and a relationship visualization unit for outputting information indicating the relationship between the classifications on the basis of the distances between the representative points.

Description

Defect analysis device, defect analysis method, and program

[0001] The present disclosure relates to a defect analysis apparatus, a defect analysis method, and a program.

[0002] A technique for classifying images based on a distance metric between images learned using deep learning is used. For example, Patent Literature 1 discloses a user interface that discriminates users based on a distance metric learned from their facial images, thereby conducting a dialogue according to the user's preferences.

[0003] International Publication No. 2012 / 88627

[0004] However, conventional image classification techniques have the problem of difficulty in classifying defects that occur on the surface of an object. For example, because the features that make up a human face are generally the same, it is relatively easy to improve the classification accuracy of face images. However, defects that occur on the surface of an object are diverse, and classification is often difficult even when judged by humans. Furthermore, when only the classification results of the classified defects are presented, it is difficult to understand the similarity or inaccuracy between the classifications.

[0005] In view of the above technical problems, the present disclosure aims to analyze the classification of defects based on images of the defects.

[0006] The present disclosure has the following configuration.

[0007] [1] A defect analysis device comprising: a model storage unit configured to store a feature extraction model that extracts, from an input image, image feature amounts that shorten the distance between similar images, the model extraction model having learned based on learning data in which defect images obtained by photographing defects occurring on a surface of an object are assigned information representing the classification of the defects; a representative point determination unit configured to determine a representative point for each classification based on the image feature amounts extracted from the defect images using the feature extraction model; and a relationship visualization unit configured to output information representing the relationship between the classifications based on the distance between the representative points.

[0008] [2] The defect analysis device according to [1] above, further comprising: an image input unit configured to receive input of a verification image obtained by photographing a surface of an object; a feature extraction unit configured to generate, as verification features, the image features extracted from the verification image using the feature extraction model; and a classification estimation unit configured to estimate the classification of the verification image based on a distance between the verification features and the representative point.

[0009] [3] The defect analysis device according to [2] above, further comprising a similar image extraction unit configured to extract the defect image similar to the verification image from the learning data based on the distance between the image features.

[0010] [4] The defect analysis device according to any one of [1] to [3] above, further comprising: a classification aggregation unit configured to determine an integrated classification that includes a plurality of classifications whose representative points are close to each other; and a model learning unit configured to re-learn the feature extraction model based on the learning data to which information representing the integrated classification has been assigned.

[0011] [5] The defect analysis device according to any one of [1] to [4] above, wherein the information representing the relationship is a heat map in which the distance between the representative points is color-coded according to the length, a scatter plot in which the representative points are arranged based on multidimensional scaling, or a dendrogram in which the representative points are hierarchically organized based on hierarchical clustering.

[0012] [6] The defect analysis apparatus according to [2] above, wherein the classification estimation unit is configured to estimate the classification of the verification image by comparing a distance between the verification feature and the representative point with a threshold.

[0013] [7] The defect analysis device according to [6] above, wherein the classification estimation unit is configured to estimate that the verification image falls into a plurality of the classifications when there are a plurality of classifications for which the distance is equal to or less than the threshold.

[0014] [8] The defect analysis device according to [6] above, wherein the classification estimation unit is configured to estimate that the verification image belongs to a new classification different from the classification when there is no classification for which the distance is equal to or less than the threshold.

[0015] [9] A defect analysis method in which a defect analysis device executes the steps of: storing in a model storage unit a feature extraction model that extracts image features from input images that bring similar images closer together, the feature extraction model having been learned based on learning data in which defect images obtained by photographing defects occurring on the surface of an object are given information representing the classification of the defects; determining a representative point for each classification based on the image features extracted from the defect images using the feature extraction model; and outputting information representing the relationship between the classifications based on the distance between the representative points.

[0016]

[10] A program for causing a computer to execute the steps of: storing in a model storage unit a feature extraction model that extracts, from an input image, image feature amounts that bring similar images closer together, the feature extraction model having been learned based on learning data in which defect images obtained by photographing defects occurring on the surface of an object are given information representing the classification of the defects; determining a representative point for each classification based on the image feature amounts extracted from the defect images using the feature extraction model; and outputting information representing the relationship between the classifications based on the distance between the representative points.

[0017] According to one aspect of the present disclosure, the classification of a defect can be analyzed based on an image of the defect.

[0018] FIG. 1 is a block diagram showing an example of the overall configuration of a defect analysis system. FIG. 2 is a block diagram showing an example of the hardware configuration of a computer. FIG. 3 is a block diagram showing an example of the functional configuration of the defect analysis system. FIG. 4 is a flowchart showing an example of the procedure for relationship visualization processing. FIG. 5 is a diagram showing an example of a defect image. FIG. 6 is a diagram showing an example of a heat map in the relationship visualization processing. FIG. 7 is a diagram showing an example of a scatter plot in the relationship visualization processing. FIG. 8 is a diagram showing an example of a dendrogram in the relationship visualization processing. FIG. 9 is a flowchart showing an example of the procedure for image classification processing. FIG. 10 is a diagram showing an example of similar images. FIG. 11 is a diagram showing an example of a heat map in the image classification processing. FIG. 12 is a diagram showing an example of a scatter plot in the image classification processing. FIG. 13 is a diagram showing an example of a dendrogram in the image classification processing. FIG. 14 is a flowchart showing an example of the procedure for model re-learning processing.

[0019] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functional configuration are designated by the same reference numerals, and redundant description will be omitted.

[0020] [Embodiment] Defects may occur on the surface of an object due to various causes. In order to identify the cause of the defect, photographed images of the defect are analyzed. For example, if a defect occurs during a product manufacturing process, there is a possibility that a defect may be hidden within the manufacturing process. Therefore, when a defect is discovered in a product during an inspection process, the defect in the product is photographed and used as a clue to identify the defect in the manufacturing process.

[0021] Conventional image classification techniques classify each image by assigning it a unique class. However, defects that occur on the surface of objects are diverse, and classification is often difficult even for humans. Conventional image classification techniques can have low classification accuracy for intermediate defects that resemble multiple classes or new defects that differ from any predefined class. On the other hand, subtle differences in defects can sometimes provide important information for identifying the cause of the defect. Therefore, analyzing the relationship between defect classifications can potentially provide useful information regarding object handling.

[0022] One embodiment of the present disclosure is a defect analysis system that classifies images (hereinafter also referred to as "defect images") of defects occurring on the surface of an object and analyzes the relationship between the defect classifications. An example of an object in this embodiment is an article having a mirror surface, such as an optical product. Another example of an object in this embodiment is a single crystal substrate, such as a semiconductor wafer. The object in this embodiment is not limited to these, and may also be, for example, an article that is mirror-finished, such as an aluminum substrate for a photosensitive drum used in a photosensitive drum of a laser printer.

[0023] In this embodiment, the defect is a linear or planar scratch on the surface of an object, or a stain or foreign matter attached to the surface of an object.

[0024] The defect analysis system of this embodiment uses pre-collected training data to train a feature extraction model that extracts image features from an input image. The training data is data in which defect images used for model training (hereinafter also referred to as "training images") are assigned information representing defect classifications (hereinafter also referred to as "defect labels").

[0025] The feature extraction model in this embodiment is trained using a metric learning technique, in which images are placed at a single point in a high-dimensional embedding space so that similar images are close to each other.

[0026] The defect analysis system in this embodiment uses a trained feature extraction model to extract image features from training images and determine representative points for each defect classification indicated by the defect label. The defect analysis system outputs information representing the classification relationships based on the distances between the representative points of each classification. Examples of the information representing the classification relationships include a heat map, a scatter diagram, a dendrogram, or the like that represent the classification relationships. The information representing the classification relationships is not limited to these, and various representation formats that are easy for a user of the defect analysis system to understand can be used.

[0027] Furthermore, the defect analysis system of this embodiment uses the trained feature extraction model to extract image features from a defect image (hereinafter also referred to as a "verification image") whose defect classification has not been specified, and estimates the classification of the defect captured in the verification image based on the distance from the representative point of each classification. At this time, images similar to the verification image (hereinafter also referred to as a "similar image") may be extracted from training images to which defect labels indicating the same classification as the estimated classification have been assigned, and output together with the estimation result.

[0028] Furthermore, the defect analysis system of this embodiment creates a new classification (hereinafter also referred to as a "new classification") when it is determined that the verification image does not fall into any classification. Furthermore, when there are multiple classifications in which the distance between representative points is short, the defect analysis system aggregates those classifications into a new classification (hereinafter also referred to as an "integrated classification"). The defect analysis system then retrains the feature extraction model using the training data to which the new classification or integrated classification has been assigned.

[0029] <Overall Configuration of Defect Analysis System> First, the overall configuration of the defect analysis system in this embodiment will be described with reference to Fig. 1. Fig. 1 is a block diagram showing an example of the overall configuration of the defect analysis system in this embodiment.

[0030] 1, a defect analysis system 1 in this embodiment includes a defect analysis device 10, an image acquisition device 20, and a user terminal 30. The defect analysis device 10, the image acquisition device 20, and the user terminal 30 are connected to each other so as to be able to communicate data with each other via a communication network N1 such as a local area network (LAN) or the Internet.

[0031] The defect analysis apparatus 10 is an information processing device such as a personal computer, workstation, or server that analyzes defect images of defects occurring on the surface of an object in response to a request from the user terminal 30. The defect analysis apparatus 10 receives defect images to be analyzed from the user terminal 30. The defect analysis apparatus 10 analyzes the received defect images and transmits the analysis results to the user terminal 30.

[0032] The image acquisition device 20 is an optical device that acquires defect images by photographing defects occurring on the surface of an object. The image acquisition device 20 may be a digital camera that captures still images or a video camera that captures moving images. The image acquisition device 20 may also be an information processing device such as a personal computer connected to various cameras, or a surface inspection device equipped with various cameras.

[0033] The user terminal 30 is an information processing terminal operated by a user, such as a personal computer, a tablet terminal, a smartphone, etc. In response to a user's operation, the user terminal 30 acquires defect images to be analyzed from the image acquisition device 20 and transmits them to the defect analysis apparatus 10. The user terminal 30 also receives analysis results from the defect analysis apparatus 10 and outputs them to the user.

[0034] 1 is merely an example, and various system configurations are possible depending on the application and purpose. For example, the defect analysis apparatus 10 may be implemented using multiple computers or as a cloud computing service. Furthermore, the defect analysis system 1 may be implemented using a standalone information processing device that combines the functions of the defect analysis apparatus 10, the image acquisition apparatus 20, and the user terminal 30.

[0035] <Hardware Configuration of Defect Analysis System> Next, the hardware configuration of the defect analysis system 1 in this embodiment will be described with reference to FIG.

[0036] <Hardware Configuration of Computer> The defect analysis device 10, the image acquisition device 20, and the user terminal 30 in this embodiment are realized by, for example, a computer. Fig. 2 is a block diagram showing an example of the hardware configuration of a computer 500 in this embodiment.

[0037] 2, the computer 500 includes a CPU (Central Processing Unit) 501, a ROM (Read Only Memory) 502, a RAM (Random Access Memory) 503, a HDD (Hard Disk Drive) 504, an input device 505, a display device 506, a communication I / F (Interface) 507, and an external I / F 508. The CPU 501, the ROM 502, and the RAM 503 form a so-called computer. The hardware components of the computer 500 are connected to each other via a bus line 509. The input device 505 and the display device 506 may be connected to the external I / F 508 for use.

[0038] The CPU 501 is a computing device that reads programs and data from a storage device such as the ROM 502 or the HDD 504 onto the RAM 503 and executes processing to realize overall control and functions of the computer 500 .

[0039] The ROM 502 is an example of a non-volatile semiconductor memory (storage device) that can retain programs and data even when the power is turned off. The ROM 502 functions as a main storage device that stores various programs, data, etc. required for the CPU 501 to execute various programs installed in the HDD 504. Specifically, the ROM 502 stores boot programs such as a Basic Input / Output System (BIOS) and an Extensible Firmware Interface (EFI) that are executed when the computer 500 starts up, as well as data such as OS (Operating System) settings and network settings.

[0040] The RAM 503 is an example of a volatile semiconductor memory (storage device) in which programs and data are erased when the power is turned off. The RAM 503 is, for example, a dynamic random access memory (DRAM) or a static random access memory (SRAM). The RAM 503 provides a working area in which various programs installed in the HDD 504 are expanded when executed by the CPU 501.

[0041] The HDD 504 is an example of a non-volatile storage device that stores programs and data. The programs and data stored in the HDD 504 include an OS, which is basic software that controls the entire computer 500, and applications that provide various functions on the OS. Note that the computer 500 may use a storage device that uses flash memory as a storage medium (e.g., an SSD (Solid State Drive)) instead of the HDD 504.

[0042] The input device 505 includes a touch panel, operation keys and buttons, a keyboard and mouse, a microphone for inputting sound data such as voice, and the like, which are used by the user to input various signals.

[0043] The display device 506 is composed of a display such as a liquid crystal display or organic electroluminescence (EL) display for displaying a screen, a speaker for outputting sound data such as voice, and the like.

[0044] The communication I / F 507 is an interface that connects to a communication network and enables the computer 500 to perform data communication.

[0045] The external I / F 508 is an interface with external devices, such as a drive device 510.

[0046] The drive device 510 is a device for loading a recording medium 511. The recording medium 511 here includes media that record information optically, electrically, or magnetically, such as CD-ROMs, flexible disks, and magneto-optical disks. The recording medium 511 may also include semiconductor memories that record information electrically, such as ROMs and flash memories. This allows the computer 500 to read from and / or write to the recording medium 511 via the external I / F 508.

[0047] The various programs to be installed in the HDD 504 are installed, for example, by setting the distributed recording medium 511 in a drive device 510 connected to the external I / F 508 and reading the various programs recorded on the recording medium 511 by the drive device 510. Alternatively, the various programs to be installed in the HDD 504 may be installed by being downloaded via the communication I / F 507 from a network different from the communication network.

[0048] <Functional Configuration of Defect Analysis System> Next, the functional configuration of the defect analysis system in this embodiment will be described with reference to Fig. 3. Fig. 3 is a block diagram showing an example of the functional configuration of the defect analysis system 1 in this embodiment.

[0049] <Functional Configuration of Defect Analysis Apparatus> As shown in FIG. 3 , the defect analysis apparatus 10 in this embodiment includes an image input unit 101, a learning data storage unit 102, a model learning unit 103, a model storage unit 104, a feature extraction unit 105, a representative point determination unit 106, a relationship visualization unit 107, a classification estimation unit 108, a similar image extraction unit 109, and a classification aggregation unit 110.

[0050] The image input unit 101, model learning unit 103, feature extraction unit 105, representative point determination unit 106, relationship visualization unit 107, classification estimation unit 108, similar image extraction unit 109, and classification aggregation unit 110 are realized by processing that is executed by the CPU 501 based on a program loaded from the HDD 504 onto the RAM 503 shown in Fig. 2. The learning data storage unit 102 and model storage unit 104 are realized by the HDD 504 shown in Fig. 2.

[0051] The image input unit 101 receives defect images from the user terminal 30. The defect images received by the image input unit 101 include training images to which defect labels have been assigned and verification images to which no defect labels have been assigned.

[0052] The training data storage unit 102 stores training images received by the image input unit 101 as training data. The training data is data in which defect labels are added to training images. The number of training data may be a sufficient amount for training a feature extraction model. The amount of training data sufficient for training a feature extraction model varies depending on the type of model, but is, for example, about 100 images for each classification.

[0053] The model training unit 103 trains a feature extraction model using the training data stored in the training data storage unit 102. The feature extraction model is a machine learning model that extracts, from an input image, image features that reduce the distance between similar images. The model training unit 103 may train the feature extraction model by applying techniques such as transfer learning or fine tuning to a pre-trained image classification model. An example of the feature extraction model is an image classification model that uses deep learning such as Alex-Net or Res-Net. The image feature is, for example, a 256-dimensional feature vector.

[0054] The model storage unit 104 stores the feature extraction model learned by the model learning unit 103 .

[0055] The feature extraction unit 105 extracts image features from the training images or the verification images using a feature extraction model stored in the model storage unit 104. Hereinafter, the image features extracted from the training images will be referred to as "training features." Also, the image features extracted from the verification images will be referred to as "verification features."

[0056] The representative point determination unit 106 determines a representative point for each defect classification based on the learning features extracted by the feature extraction unit 105. The representative point for a classification is a coordinate that represents a set of learning features corresponding to training images to which the same defect label has been assigned when the learning features are arranged in an embedding space. An example of a representative point is the mean of the feature vectors. Another example of a representative point is the center of gravity of the set of feature vectors. The representative point is not limited to these, and may be determined in any way as long as it represents the set of learning features for each classification.

[0057] The relationship visualization unit 107 generates information representing the relationship between the classifications based on the distance between the representative points of each classification, and transmits the information to the user terminal 30. An example of the distance is cosine similarity. Other examples of the distance are Euclidean distance or Mahalanobis distance. The distance is not limited to these distance measures, and any distance measure according to the feature vector can be used.

[0058] A heat map is a matrix in which the distance between representative points is color-coded according to the distance between them. A scatter plot is a graph in which representative points of each classification are arranged on a two-dimensional plane based on, for example, multidimensional scaling (MDS). A dendrogram is a tree diagram in which representative points of each classification are hierarchically organized in a tree structure based on, for example, hierarchical clustering.

[0059] The classification estimation unit 108 estimates the classification of the defect captured in the verification image based on the distance between the verification feature extracted by the feature extraction unit 105 and the representative point of each classification. The classification estimation unit 108 estimates the classification of the defect captured in the verification image by comparing the distance between the verification feature and the representative point with a predetermined threshold.

[0060] The classification estimation unit 108 may assign a defect label representing the estimated classification to the verification image and store the defect label in the learning data storage unit 102. In this way, the verification image with the estimated defect classification is added as new learning data.

[0061] The similar image extraction unit 109 extracts similar images that are similar to the verification image from the training images stored in the training data storage unit 102 based on the distance between each training feature and the verification feature.

[0062] When there are multiple classifications in which the distance between representative points is equal to or less than a predetermined threshold, the classification aggregator 110 determines an integrated classification that includes these classifications. When determining the integrated classification, the classification aggregator 110 updates, in the learning data stored in the learning data storage unit 102, defect labels that represent the multiple classifications included in the integrated classification to defect labels that represent the integrated classification.

[0063] <Functional Configuration of Image Acquisition Device> As shown in FIG. 3 , the image acquisition device 20 in this embodiment includes an imaging unit 201 and an image storage unit 202 .

[0064] The photographing unit 201 is realized by a camera connected to the external I / F 508 shown in Fig. 2. The image storage unit 202 is realized by the HDD 504 shown in Fig. 2.

[0065] The photographing unit 201 photographs a defect occurring on the surface of an object and generates a defect image. The photographing unit 201 may photograph a still image or may photograph a video and extract an image in which the defect is reflected.

[0066] The image storage unit 202 stores the defect image captured by the imaging unit 201 .

[0067] <Functional Configuration of User Terminal 30> As shown in FIG. 3, the user terminal 30 in this embodiment includes an image acquisition unit 301, a classification unit 302, an image transmission unit 303, and a result display unit 304.

[0068] The image acquisition unit 301, classification unit 302, and image transmission unit 303 are realized by processing that is executed by the CPU 501 in accordance with a program loaded from the HDD 504 onto the RAM 503 shown in Fig. 2. The result display unit 304 is realized by the display device 506 shown in Fig. 2.

[0069] The image acquisition unit 301 acquires a defect image from the image acquisition device 20 in response to a user request.

[0070] The classification unit 302 generates learning images by assigning defect labels to defect images in response to user operations. The defect images to which the defect labels are assigned may be selected only for those defect images for which the user can identify the defect classification.

[0071] The image transmission unit 303 transmits the learning image or the verification image to the defect analysis apparatus 10 in response to a user operation.

[0072] The result display unit 304 receives information indicating the classification relationship from the defect analysis device 10 and outputs it to the display device 506 or the like.

[0073] <Processing Procedure of Defect Analysis System> Next, the processing procedure of the defect analysis method executed by the defect analysis system 1 in this embodiment will be described with reference to Fig. 4 to Fig. 14. The defect analysis method in this embodiment includes a relationship visualization process (see Fig. 4), an image classification process (see Fig. 9), and a model re-learning process (see Fig. 14).

[0074] 4 is a flowchart showing an example of the procedure of the relationship visualization process according to this embodiment. The relationship visualization process is a process in which a feature extraction model is trained using training images, and the relationship between defect classifications is visualized based on image features extracted from the training images.

[0075] In step S1, the photographing unit 201 included in the image acquisition device 20 photographs a defect occurring on the surface of an object and generates a defect image. Next, the photographing unit 201 stores the generated defect image in the image storage unit 202. The image storage unit 202 accumulates a plurality of defect images.

[0076] In step S2, in response to a user operation, the image acquisition unit 301 included in the user terminal 30 transmits a learning image acquisition request to the image acquisition device 20. In response to the learning image acquisition request, the image acquisition device 20 transmits a plurality of defect images stored in the image storage unit 202 to the user terminal 30. The image acquisition unit 301 sends the plurality of defect images received from the image acquisition device 20 to the classification unit 302.

[0077] In step S3, the classifier 302 included in the user terminal 30 receives the defect images from the image acquirer 301. Next, the classifier 302 generates a plurality of learning images by assigning defect labels to the defect images in response to a user operation. Subsequently, the classifier 302 sends the generated learning images to the image transmitter 303.

[0078] The defect classification can be set arbitrarily by the user. The defect label is data indicating the code value assigned to the defect classification. There are no restrictions on the types of defect classification, and the user can set as many classifications as they deem necessary for analysis.

[0079] In this embodiment, the classifications are 0: scratches, 1: marker marks, 2: vertical lines, 3: surface stains, 4: blurred, 5: mottled, 6: short fibers, 7: long fibers, 8: sagging, 9: thin horizontal lines, 10: thick horizontal lines, 11: oval dots, 12: thick black dots, 13: small haze, and 14: small black dots. Note that the number before the colon (:) is a code value assigned to the classification, and the character string after the colon (:) is the name of the classification.

[0080] Any code value can be assigned to a classification as long as it does not overlap with other classifications. The name of the classification can be arbitrarily determined so that it is easy for users to understand.

[0081] Fig. 5 is a diagram showing an example of a defect image. As shown in Fig. 5, there can be a wide variety of defects that occur on the surface of an object. Fig. 5 shows examples of defect images corresponding to 0: scratch, 3: surface stain, 2: vertical line, 8: sagging, 6: short fiber, 7: long fiber, 9: thin horizontal line, 10: thick horizontal line, 11: elliptical dot, 12: thick black dot, 13: small haze, and 14: small black dot.

[0082] The example in Figure 5 shows an image in which the defect's characteristics are clear, but intermediate defects can also occur, making it difficult to determine which category they fall into. For example, when there is a long horizontal line scratch, it can be difficult to determine whether it should be classified as a thin horizontal line (9) or a thick horizontal line (10). Also, when there is a linear black defect, it can be difficult to determine whether it is a long fiber (7) or a thin horizontal line (9). Furthermore, a single image may contain multiple defects of different types.

[0083] The user may select only defect images that clearly belong to a specific classification from the acquired defect images and assign defect labels to them. Defect images to which the user has not assigned a defect label may be used as verification images in the image classification process.

[0084] 4 , in step S4, the image transmission unit 303 included in the user terminal 30 receives the plurality of training images from the classification unit 302. Next, the image transmission unit 303 transmits the plurality of training images to the defect analysis apparatus 10 in response to a user operation.

[0085] In step S5, the image input unit 101 included in the defect analysis apparatus 10 receives a plurality of training images from the user terminal 30. Next, the image input unit 101 stores the received plurality of training images in the training data storage unit 102 as training data.

[0086] In step S6, the model learning unit 103 included in the defect analysis apparatus 10 reads out the learning data from the learning data storage unit 102. Next, the model learning unit 103 uses the read-out learning data to learn a feature extraction model. Subsequently, the model learning unit 103 stores the learned feature extraction model in the model storage unit 104.

[0087] In step S7, the feature extraction unit 105 included in the defect analysis apparatus 10 reads out the feature extraction model stored in the model storage unit 104. Next, the feature extraction unit 105 reads out training images from the training data storage unit 102 for each defect classification indicated by the defect label.

[0088] Next, the feature extraction unit 105 generates training features by inputting the training images for each defect classification into a feature extraction model. The feature extraction unit 105 sends the generated training features for each defect classification to the representative point determination unit 106.

[0089] In step S8, the representative point determination unit 106 included in the defect analysis apparatus 10 receives the learned features for each defect classification from the feature extraction unit 105. Next, the representative point determination unit 106 arranges the learned features in the embedding space. Subsequently, the representative point determination unit 106 determines a representative point for each defect classification based on the embedding space in which the learned features have been arranged. The representative point determination unit 106 sends information representing the determined representative point to the relationship visualization unit 107.

[0090] In step S9, the relationship visualization unit 107 included in the defect analysis apparatus 10 receives information representing the representative points from the representative point determination unit 106. Next, the relationship visualization unit 107 calculates the distance between the representative points of each classification. Subsequently, the relationship visualization unit 107 generates information representing the classification relationships based on the distances between the representative points. The relationship visualization unit 107 transmits the information representing the classification relationships to the user terminal 30.

[0091] The relationship visualization unit 107 can represent the classification relationships in various ways. A first example of information representing the classification relationships is a heat map representing the classification relationships. A second example of information representing the classification relationships is a scatter plot representing the classification relationships. A third example of information representing the classification relationships is a dendrogram representing the classification relationships.

[0092] (Heat Map) Fig. 6 is a diagram showing an example of a heat map in the relationship visualization process. As shown in Fig. 6, the heat map in the relationship visualization process is a matrix in which the distances between representative points are color-coded according to the length.

[0093] In the example of Fig. 6, the distance between representative points is expressed as cosine similarity, and the distance is color-coded for each combination of classifications. The closer the distance is to 0, the more similar the defect classifications are, and the closer the distance is to 2, the more dissimilar the defect classifications are. In Fig. 6, the closer the distance is, the darker the color is. The user can understand the similarity relationship between classifications by observing the shade of color in the heat map.

[0094] (Scatter Plot) Fig. 7 is a diagram showing an example of a scatter plot in the relationship visualization process. As shown in Fig. 7, the scatter plot in the relationship visualization process is a graph in which representative points of each classification are arranged on a two-dimensional plane based on multidimensional scaling. By observing the distances between the representative points in the scatter plot, the user can grasp the similarity relationships between the classifications.

[0095] (Dendrogram) Fig. 8 is a diagram showing an example of a dendrogram in the relationship visualization process. As shown in Fig. 8, the dendrogram in the relationship visualization process is a tree diagram in which the representative points of each classification are hierarchically arranged in a tree structure based on hierarchical clustering. The user can understand the similarity relationships between classifications by observing the lengths of the paths connecting the classifications in the dendrogram and the heights of the branches.

[0096] 4, in step S10, the result display unit 304 included in the user terminal 30 receives information indicating the classification relationships from the defect analysis apparatus 10. Next, the result display unit 304 displays the information indicating the classification relationships on the display device 506.

[0097] 9 is a flowchart showing an example of the procedure of the image classification process according to this embodiment. The image classification process is a process of estimating the classification of a defect captured in a verification image using a trained feature extraction model and visualizing the relationship between the verification image and the classification of the defect.

[0098] In step S11, the photographing unit 201 included in the image acquisition device 20 photographs a defect occurring on the surface of an object and generates a defect image. Next, the photographing unit 201 stores the generated defect image in the image storage unit 202. The image storage unit 202 accumulates a plurality of defect images.

[0099] In step S12, in response to a user operation, the image acquisition unit 301 included in the user terminal 30 transmits a request to acquire a verification image to the image acquisition device 20. In response to the request to acquire a verification image, the image acquisition device 20 transmits the defect image stored in the image storage unit 202 to the user terminal 30. The image acquisition unit 301 sends the defect image received from the image acquisition device 20 to the image transmission unit 303.

[0100] In step S13, the image transmission unit 303 included in the user terminal 30 receives the defect image from the image acquisition unit 301. Next, the image transmission unit 303 transmits a verification image selected from the defect image to the defect analysis apparatus 10 in response to a user operation.

[0101] In step S14, the image input unit 101 included in the defect analysis apparatus 10 receives the verification image from the user terminal 30. Next, the image input unit 101 sends the received verification image to the feature extraction unit 105.

[0102] In step S15, the feature extraction unit 105 included in the defect analysis apparatus 10 receives the verification image from the image input unit 101. Next, the feature extraction unit 105 reads out the feature extraction model stored in the model storage unit 104. Subsequently, the feature extraction unit 105 inputs the verification image into the feature extraction model to generate verification features. The feature extraction unit 105 sends the generated verification features to the classification estimation unit 108.

[0103] In step S16, the classification estimation unit 108 included in the defect analysis apparatus 10 receives the verification feature from the feature extraction unit 105. Next, the classification estimation unit 108 acquires the representative point of each classification from the representative point determination unit 106. Subsequently, the classification estimation unit 108 arranges the verification feature and the representative point of each classification in the embedding space. Next, the classification estimation unit 108 calculates the distance between the verification feature and the representative point of each classification. Then, the classification estimation unit 108 estimates the classification of the defect captured in the verification image based on the distance between the verification feature and the representative point of each classification.

[0104] The classification estimation unit 108 estimates the classification of the defect captured in the verification image by comparing the distance between the verification feature and the representative point of each classification with a predetermined threshold. Note that the threshold used for estimation may be set to an arbitrary value by a user operation.

[0105] The category estimation unit 108 determines that the verification image belongs to a category when there is one representative point whose distance from the verification feature is equal to or less than a threshold. The category estimation unit 108 determines that the verification image belongs to a plurality of categories when there are multiple representative points whose distance from the verification feature is equal to or less than a threshold. The category estimation unit 108 determines that the verification image belongs to a new category when there is no representative point whose distance from the verification feature is equal to or less than a threshold.

[0106] The classification estimation unit 108 sends the verification feature and the estimation result indicating the estimated classification to the similar image extraction unit 109. The classification estimation unit 108 may assign a defect label indicating the estimated classification to the verification image and add it to the learning data stored in the learning data storage unit 102.

[0107] In step S17, the similar image extraction unit 109 included in the defect analysis apparatus 10 receives the verification feature and the estimation result from the classification estimation unit 108. Next, the similar image extraction unit 109 reads out a training image to which a defect label indicating the same classification as the estimation result has been assigned from the training data storage unit 102. Next, the similar image extraction unit 109 calculates the distance between the training feature and the verification feature corresponding to the read training image.

[0108] The similar image extraction unit 109 extracts similar images from the training images based on the distance between each training feature and the verification feature. The similar image extraction unit 109 may extract all training images whose distance from the verification feature is equal to or less than a predetermined threshold, or may extract a predetermined number of training images in order of shortest distance from the verification feature. The similar image extraction unit 109 sends the verification feature, the estimation result, and the similar images to the relationship visualization unit 107.

[0109] FIG. 10 is a diagram showing an example of a similar image. The example in FIG. 10 shows the result of extracting three similar images that are closest to the verification image in ascending order of distance. In the example in FIG. 10, a thin horizontal line extending horizontally from near the center is captured in the verification image. Similar images 1 to 3 also contain horizontal lines of approximately the same thickness as the verification image, indicating that they are similar to the verification image. As shown in FIG. 10, similar images may be provided with information indicating the distance from the verification image.

[0110] 9 , in step S18, the relationship visualization unit 107 included in the defect analysis apparatus 10 receives the verification feature, the estimation result, and the similar image from the similar image extraction unit 109. Next, the relationship visualization unit 107 generates information representing the relationship between the verification image and the classification based on the distance between the verification feature and the representative point of each classification. The relationship visualization unit 107 transmits the information representing the relationship between the verification image and the classification to the user terminal 30.

[0111] The relationship visualization unit 107 can represent the relationship between the verification image and the classification in the same way as the method for representing the relationship between classifications in the relationship visualization process. That is, the relationship visualization unit 107 can represent the relationship between the verification image and the classification using a heat map, a scatter diagram, or a dendrogram.

[0112] (Heat Map) Fig. 11 is a diagram showing an example of a heat map in image classification processing. As shown in Fig. 11, the heat map in image classification processing is a matrix in which the verification image is designated as "15: input image" and the distance from the representative point of each classification is color-coded according to the length. By observing the color shading in the heat map, the user can understand the similarity relationship between the verification image and each classification.

[0113] (Scatter Plot) Fig. 12 is a diagram showing an example of a scatter plot in image classification processing. As shown in Fig. 12, the scatter plot in image classification processing is a graph in which the verification image is designated as "15: Input Image" and the verification features and representative points of each classification are arranged on a two-dimensional plane based on multidimensional scaling. By observing the distance between the input image and the representative points of each classification in the scatter plot, the user can grasp the similarity relationship between the verification image and each classification.

[0114] (Dendrogram) Fig. 13 is a diagram showing an example of a dendrogram in image classification processing. As shown in Fig. 13, the dendrogram in image classification processing is a tree diagram in which the verification image is "15: input image" and the input image and representative points of each classification are hierarchically arranged in a tree structure based on hierarchical clustering. By observing the length of the paths connecting the input image and each classification in the dendrogram and the height of the branches, the user can understand the similarity relationship between the verification image and each classification.

[0115] 9 , in step S19, the result display unit 304 included in the user terminal 30 receives information indicating the relationship between the verification image and the classification from the defect analysis apparatus 10. Next, the result display unit 304 displays the information indicating the relationship between the verification image and the classification on the display device 506.

[0116] 14 is a flowchart showing an example of the procedure of the model re-learning process according to this embodiment. The model re-learning process is a process of reorganizing defect classifications based on analysis results and re-learning a feature extraction model using training data to which the reorganized classifications have been assigned.

[0117] In step S21, when the classification estimation unit 108 provided in the defect analysis device 10 determines that the verification image corresponds to a new classification in the image classification process, it assigns a defect label representing the new classification to the verification image and adds it to the learning data stored in the learning data storage unit 102.

[0118] In step S22, the classification aggregating unit 110 included in the defect analysis apparatus 10 acquires the representative points of each classification from the representative point determining unit 106. Next, the classification aggregating unit 110 arranges the representative points of each classification in the embedding space. Subsequently, based on the embedding space in which the representative points of each classification are arranged, the classification aggregating unit 110 determines whether there are multiple classifications whose representative points are spaced apart by a predetermined threshold or less. If there are multiple classifications whose representative points are spaced apart by a threshold or less, the classification aggregating unit 110 determines an integrated classification that includes these classifications. The integrated classification may select one classification from the multiple classifications, or may generate a new classification.

[0119] The threshold value used for the determination by the classification aggregation unit 110 may be the same as or different from the threshold value used for estimation by the classification estimation unit 108. The threshold value used for the determination may be set to an arbitrary value by a user operation, similar to the threshold value used for estimation.

[0120] In step S23, the classification aggregator 110 included in the defect analysis apparatus 10 reads out, from the learning data storage unit 102, learning data to which defect labels representing the multiple classifications included in the integrated classification have been assigned. Next, the classification aggregator 110 updates the defect labels assigned to the read learning data to defect labels representing the integrated classification. Subsequently, the classification aggregator 110 stores the learning data to which the defect labels representing the integrated classification have been assigned in the learning data storage unit 102.

[0121] In step S24, the model learning unit 103 included in the defect analysis apparatus 10 reads out learning data from the learning data storage unit 102. Next, the model learning unit 103 reads out the feature extraction model stored in the model storage unit 104. Subsequently, the model learning unit 103 re-learns the feature extraction model using the read-out learning data. Then, the model learning unit 103 stores the re-learned feature extraction model in the model storage unit 104.

[0122] <Effects of the Embodiment> The defect analysis system of the present embodiment trains a feature extraction model based on training data in which defect images are assigned defect classifications. The feature extraction model is trained to extract image features from input images that reduce the distance between similar images. The defect analysis system outputs information indicating the relationship between defect classifications based on the distance between representative points determined from the image features of the defect images. Therefore, the defect analysis system of the present embodiment can analyze the classification of defects based on captured images of the defects.

[0123] In particular, the defect analysis system of this embodiment can visualize the relationship between defect classifications in various representation formats, such as a heat map that visualizes the distances between representative points, a scatter diagram, a dendrogram, etc. Therefore, the defect analysis system of this embodiment can output analysis results that are highly convincing to the user.

[0124] Furthermore, the defect analysis system of this embodiment estimates the classification of a defect captured in a verification image, whose classification is not specified, based on the image feature of the verification image and the distance between the representative point of each classification. At this time, an image similar to the verification image may be extracted from the defect images included in the estimated classification and output together with the estimation result. Therefore, the defect analysis system of this embodiment can output a convincing estimation result for an intermediate defect that resembles any of multiple classifications or a new defect that does not belong to any of the predetermined classifications.

[0125] Furthermore, the defect analysis system of this embodiment retrains the feature extraction model using training data to which a new classification, in which the verification image does not fall into any classification, or an integrated classification, in which multiple classifications with short distances between representative points, are assigned. Using the retrained feature extraction model, it is possible to analyze the relationships between the classifications of defects reorganized based on the analysis results. Therefore, the defect analysis system of this embodiment can acquire defect classifications appropriate for the user's usage environment and output analysis results that are more convincing to the user.

[0126] [Supplementary Note] Each function of the above-described embodiments can be realized by one or more processing circuits. Here, the term "processing circuit" in this specification includes a processor programmed to execute each function by software, such as a processor implemented by an electronic circuit, as well as devices such as an ASIC (Application Specific Integrated Circuit), a DSP (Digital Signal Processor), an FPGA (Field Programmable Gate Array), and conventional circuit modules designed to execute each of the above-described functions.

[0127] Although the embodiments of the present invention have been described in detail above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention described in the claims.

[0128] This application claims priority from Japanese Patent Application No. 2022-111662, filed with the Japan Patent Office on July 12, 2022, the entire contents of which are incorporated herein by reference.

[0129] 1 Defect analysis system 10 Defect analysis device 101 Image input unit 102 Learning data storage unit 103 Model learning unit 104 Model storage unit 105 Feature extraction unit 106 Representative point determination unit 107 Relationship visualization unit 108 Classification estimation unit 109 Similar image extraction unit 110 Classification aggregation unit 20 Image acquisition device 201 Photography unit 202 Image storage unit 30 User terminal 301 Image acquisition unit 302 Classification assignment unit 303 Image transmission unit 304 Result display unit

Claims

1. A model storage unit is configured to store a feature extraction model that stores image features from input images that make similar images closer together, which is trained on training data in which information representing the classification of the defects is attached to defect images taken of defects on the surface of an object. A representative point determination unit is configured to determine a representative point for each classification based on the image features extracted from the defective image using the feature extraction model, A relationship visualization unit configured to output information representing the classification relationship based on the distance between the aforementioned representative points, A defect analysis device equipped with the following features.

2. A defect analysis apparatus according to claim 1, An image input unit configured to accept verification images taken of the surface of an object, A feature extraction unit is configured to generate the image features extracted from the verification image using the feature extraction model as verification features, A classification estimation unit is configured to estimate the classification of the verification image based on the distance between the verification feature and the representative point. A defect analysis device equipped with additional features.

3. A defect analysis apparatus according to claim 2, The system further includes a similar image extraction unit configured to extract defective images similar to the verification image from the training data based on the distance between the aforementioned image features. Defect analysis device.

4. A defect analysis apparatus according to any one of claims 1 to 3, A classification aggregation unit is configured to determine an integrated classification that includes multiple classifications where the distance between the representative points is short, A model learning unit configured to retrain the feature extraction model based on the training data to which information representing the integrated classification is attached, A defect analysis device equipped with additional features.

5. A defect analysis apparatus according to any one of claims 1 to 3, The information representing the aforementioned relationship is a heatmap in which the distances between the representative points are color-coded according to their length, a scatter plot in which the representative points are arranged based on multidimensional scaling, or a dendrogram in which the representative points are hierarchically arranged based on hierarchical clustering. Defect analysis device.

6. A defect analysis apparatus according to claim 2, The classification estimation unit is configured to estimate the classification of the verification image by comparing the distance between the verification feature and the representative point with a threshold. Defect analysis device.

7. A defect analysis apparatus according to claim 6, The classification estimation unit is configured to estimate that the verification image belongs to multiple classifications if there are multiple classifications for which the distance is less than or equal to the threshold. Defect analysis device.

8. A defect analysis apparatus according to claim 6, The classification estimation unit is configured to estimate that if there is no classification for which the distance is less than or equal to the threshold, the verification image corresponds to a new classification different from the aforementioned classification. Defect analysis device.

9. The defect analysis device, A procedure for storing in a model memory unit a feature extraction model that extracts image features from input images that make similar images closer together, based on training data in which information representing the classification of the defects is attached to defect images taken of defects on the surface of an object, and A procedure for determining representative points for each classification based on the image features extracted from the defective image using the feature extraction model, A procedure for outputting information representing the classification relationship based on the distance between the aforementioned representative points, A defect analysis method that performs this analysis.

10. On the computer, A procedure for storing in a model memory unit a feature extraction model that extracts image features from input images that make similar images closer together, based on training data in which information representing the classification of the defects is attached to defect images taken of defects on the surface of an object, and A procedure for determining representative points for each classification based on the image features extracted from the defective image using the feature extraction model, A procedure for outputting information representing the classification relationship based on the distance between the aforementioned representative points, A program to execute.