Defect classification apparatus, method and procedure

Abstract

Embodiments of the present invention relate to a defect classification apparatus, method, and program. A defect classification apparatus, method, and program capable of supporting high-precision defect inspection are provided. The defect classification apparatus includes a first acquisition unit, a second acquisition unit, a transformation unit, a calculation unit, a detection unit, and a classification unit. The first acquisition unit acquires a first design image related to a first inspection object, the first design image being an image based on design data created using design software. The second acquisition unit acquires a first photographic image, the first photographic image being obtained by photographing the first inspection object generated based on the design data. The transformation unit transforms the first design image into a reference image expressed using a second photographic image obtained by photographing a second inspection object without defects. The calculation unit calculates the confidence level of the reference image. The detection unit compares the reference image and the first photographic image to detect defects in the first inspection object. The classification unit classifies the defects based on the confidence level.

Description

[0001] Cross-references of related applications

[0002] This application is based on and claims priority to Japanese Patent Application No. 2021-039436, filed on March 11, 2021, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The embodiments of the present invention relate to defect classification apparatus, methods, and procedures. Background Technology

[0004] In manufacturing processes based on circuit patterns of semiconductor devices, defect inspection is crucial to prevent a decrease in product yield. For example, in semiconductor devices, there is a defect inspection called Die to Database inspection, which compares a photographic image of the circuit pattern with a mask image obtained from a pseudo-photograph of the circuit pattern. This inspection simply compares two images; its principle is simple, and therefore it is installed in various inspection devices.

[0005] Furthermore, in defect inspection, it is useful to classify and judge defects by categories such as those that should be corrected, those that will not affect the operation of the transferred semiconductor device, or pseudo-defects. Generally speaking, visual classification has large deviations in accuracy due to differences in the evaluator's skill level, and it is also time-consuming. Therefore, various automated methods related to defect classification have been proposed.

[0006] For example, there is a method that overlays areas with large differences between the object image and the reference image onto the design data to classify defects that depend on the design layout. The reference image used is based on the premise of faithfully reproducing a photographic image of a mask without defects. Therefore, when the reproducibility of the reference image is low, the aforementioned differences become large, making it impossible to handle situations where defects are falsely detected. Summary of the Invention

[0007] The problem to be solved by the present invention is to provide a defect classification device, method and program that can support high-precision defect inspection.

[0008] The defect classification apparatus according to this embodiment includes a first acquisition unit, a second acquisition unit, a transformation unit, a calculation unit, a detection unit, and a classification unit. The first acquisition unit acquires a first design image related to a first inspection object, which is an image based on design data created using design software. The second acquisition unit acquires a first photographic image, obtained by photographing the first inspection object generated based on the design data. The transformation unit transforms the first design image into a reference image expressed using a second photographic image, obtained by photographing a second inspection object without defects. The calculation unit calculates the confidence level of the reference image. The detection unit compares the reference image and the first photographic image to detect defects in the first inspection object. The classification unit classifies the defects based on the confidence level.

[0009] The defect classification device based on the above structure can support high-precision defect inspection. Attached Figure Description

[0010] Figure 1 This is a block diagram illustrating the defect classification apparatus according to this embodiment.

[0011] Figure 2 It is a block diagram showing the details of the transformation section.

[0012] Figure 3 This is a concept diagram illustrating the use of a tree structure to store dictionaries in the dictionary storage section.

[0013] Figure 4 This is an example of pasting the second photographic patch image.

[0014] Figure 5 This is another example of pasting the second photographic patch image.

[0015] Figure 6 This is a flowchart illustrating an example of the operation of the defect classification device according to this embodiment.

[0016] Figure 7 This is a diagram showing the first example of the classification results for defects in the classification section.

[0017] Figure 8 This is a diagram showing the second example of the classification results for defects in the classification section.

[0018] Figure 9 This is a diagram illustrating an example of the hardware structure of a defect classification device.

[0019] Symbol Explanation

[0020] 10: Defect classification device; 350: First design patch image; 41: Sampling point; 42: Second photographic patch image; 43: Generated pixel; 101: Design image acquisition unit; 102: Photographic image acquisition unit; 103: Image storage unit; 104: Transformation unit; 105: Calculation unit; 106: Detection unit; 107: Classification unit; 108: Display unit; 302-1, 302-2, 303-1, 303-2, 303-3, 303-4, 304-1, 304-2: Representative points; 701. 801: Reference image; 702, 802: Confidence level; 703, 803: First photographic image; 704: Recognizer; 705, 804: Image; 706, 805: Defect information; 710: Defect; 901: CPU; 902: RAM; 903: ROM; 904: Storage device; 905: Display device; 906: Input device; 907: Communication device; 1041: Patch image acquisition unit; 1042: Retrieval unit; 1043: Dictionary storage unit; 1044: Reference image generation unit. Detailed Implementation

[0021] Hereinafter, the defect classification apparatus, method, and procedure according to the embodiments will be described in detail with reference to the accompanying drawings. Furthermore, in the following embodiments, portions with the same reference numerals are assumed to perform the same operations, and repeated descriptions are omitted where appropriate.

[0022] Reference Figure 1 The block diagram illustrates the defect classification device involved in the implementation method.

[0023] The defect classification device 10 according to the embodiment includes a design image acquisition unit 101, a photographic image acquisition unit 102, an image storage unit 103, a transformation unit 104, a calculation unit 105, a detection unit 106, a classification unit 107, and a display unit 108.

[0024] The design image acquisition unit 101 acquires a first design image related to the inspection object (first inspection object), based on design data generated using design software. The design software is, for example, computer-aided design software such as CAD (Computer-Aided Design). In the design process, design data is created through steps such as functional design to achieve the required functions, logic design to create circuit diagrams, and physical design to execute component placement and wiring layout. The first design image, for example, is a binary image corresponding to the presence or absence of patterns in the design data, similar to die-to-database checks. Alternatively, it can be an image based on patterns in the design data.

[0025] The photographic image acquisition unit 102 acquires a first photographic image, which is obtained by photographing the object to be inspected, which is actually generated based on the design data of the object to be inspected. The photographic device used is, for example, an optical camera. Alternatively, it could be a common electron microscope, an X-ray camera, or various cameras similar in principle. That is, light (electromagnetic waves) is scanned from a light source onto the object to be inspected, and the intensity of the transmitted or reflected light (electromagnetic waves) reaching the sensor is converted into pixel values, thereby imagering the object. The light source here is, for example, a DUV (Deep Ultraviolet) laser. Furthermore, the sensor uses a sensor arranged in a 2D array, similar to a CMOS sensor. Among these light sources and sensors, the wavelength of the light and the configuration of the sensor array are appropriately determined according to the fineness of the pattern of the object to be inspected.

[0026] Image storage unit 103 collects image pairs of the second design image and the corresponding second photographic image and stores them as a database. The second design image is a design image based on design data related to a previously inspected object (the second inspection object) that is different from the aforementioned inspection object. The second photographic image is a photographic image of a defect-free inspection object (also called a qualified product or a normal product) generated based on design data related to the second design image. "Defect-free" means, for example, that no defects are found in the photographic image by visual inspection or other appearance inspection systems.

[0027] Preferably, the second photographic image included in the image pair is an image captured using the same photographic system as that used when acquiring the first photographic image of the object being examined, but the image pair can also be an image acquired using a different photographic system. Additionally, it is preferable that the image pair has the same resolution, but the resolutions of the images constituting the image pair can also differ.

[0028] Image pairs can also be distinguished based on image production information such as the photographic system used for image acquisition, image resolution information, and information categorized according to design data patterns. The set of image pairs distinguished by category is called an image pair class. Here, the design data pattern represents geometric information such as the length of a straight line and the curvature of a curve, as well as information about the type of circuit being designed, such as analog circuits like RF circuits or digital circuits like logic operation circuits.

[0029] The transformation unit 104 receives the first design image from the design image acquisition unit 101. Referring to the image pair in the image storage unit 103, the transformation unit 104 transforms the first design image into a reference image expressed using a second photographic image in a manner that reproduces a defect-free photographic image related to the first design image. In other words, based on the first design image, the transformation unit 104 generates a reference image that is a pseudo-photographic image of the inspection object in a defect-free state. The reference image is, for example, a mask image used for defect inspection.

[0030] The calculation unit 105 calculates the confidence level of the reproducibility of the reference image generated by the transformation unit 104.

[0031] The detection unit 106 receives a reference image from the transformation unit 104 and a first photographic image from the photographic image acquisition unit 102. The detection unit 106 compares the reference image and the first photographic image to detect whether there is a defect in the object under inspection and, if there is a defect, to detect the location of the defect in the first photographic image (hereinafter also referred to as the defect location).

[0032] The classification unit 107 receives reference images from the transformation unit 104, confidence levels from the calculation unit 105, and the first photographic image and information related to the defect location from the detection unit 106. The classification unit 107 classifies the defects at the defect locations and obtains classification results. For example, in addition to defects that should be corrected (linewidth defects, hole defects), defects that will not cause problems in circuit operation and false defects that are mistakenly detected in qualified products (normal products) can be classified.

[0033] The display unit 108 receives the first photographic image, defect location, and defect classification result from the classification unit 107 and displays them on a display, for example, that is mounted on the defect classification device 10. Furthermore, the display unit 108 is not limited to display; it can also output the data of the first photographic image, defect location, and defect classification result to an external device. For example, the data of the first photographic image, defect location, and defect classification result can be sent to an external device with screen display capabilities, such as a tablet terminal or a smartphone.

[0034] Furthermore, the defect classification device 10 according to the embodiment includes an image storage unit 103, but the image storage unit 103 may also exist outside the defect classification device 10, such as an external server that can be connected to the defect classification device 10.

[0035] Next, refer to Figure 2 The block diagram illustrates the details of the transformation unit 104.

[0036] The transformation unit 104 includes a patch image acquisition unit 1041, a retrieval unit 1042, a dictionary storage unit 1043, and a reference image generation unit 1044.

[0037] The patch image acquisition unit 1041 receives the first design image from the design image acquisition unit 101 and cuts a portion of the image from the first design image. Hereinafter, the cut portion of the image is referred to as the first design patch image. As a method for cutting the first design patch image, multiple sampling points are set on the acquired first design image, and a portion of the image of a predetermined size is cut with the sampling points as the center, thereby generating the first design patch image.

[0038] The retrieval unit 1042 retrieves multiple image pairs from the image storage unit 103, and extracts the second design patch image by associating it with the second design image and the second photographic patch image by associating them with the second photographic image, respectively, and stores them as a dictionary in the dictionary storage unit 1043. As a method for retrieving multiple image pairs from the image storage unit 103, for example, all image pairs contained in the image storage unit 103 can be retrieved, or image pairs can be retrieved randomly. Furthermore, if multiple image pair classes exist, the same number of image pairs can be randomly retrieved from each of the multiple image pair classes, or only from one image pair class.

[0039] The second design patch image is a partial image cropped from the second design image, and the second photographic patch image is a partial image cropped from the second photographic image. The number of cropped partial images can be appropriately set according to the storage capacity that the dictionary storage unit 1043 can hold. In addition, regarding the cropping position, the same position in the subject is envisioned in both the second design patch image and the second photographic patch image.

[0040] The retrieval unit 1042 receives the first design patch image from the patch image acquisition unit 1041, and retrieves a second design patch image similar to the first design patch image by referring to the dictionary storage unit 1043. It then extracts a second photographic patch image that forms a pair with the similar second design patch image. When retrieving a second design patch image similar to the first design patch image, the similarity can be any index that allows calculation of the distance between vectors. For example, the arrangement of pixel values ​​in the first design patch image can be considered as a vector, and the similarity can be calculated using the L1 norm or L2 norm of the difference between the pixel values ​​of the first and second design patch images. Alternatively, the standardized cross-correlation value can be used as the similarity. In other words, any method can be used to calculate the similarity between images, such as SSD (Sum of Squared Difference) or SAD (Sum of Absolute Difference), which relates to the difference in pixel values ​​between the first and second design patch images. The retrieval unit 1042 can also extract the calculated similarity and the second photographic patch image from the dictionary storage unit 1043.

[0041] Furthermore, it is envisioned that the retrieval unit 1042 generates a dictionary related to the patch image and stores the dictionary in the dictionary storage unit 1043 before retrieving a second design patch image similar to the first design patch image, but it is not limited to this. For example, a dictionary may be generated periodically from the image storage unit 103 and stored in the dictionary storage unit 1043. Alternatively, a dictionary related to the patch image may be generated from the new image pair whenever a new image pair is stored in the image storage unit 103, and the group of patch images may be stored in the dictionary storage unit 1043.

[0042] The dictionary storage unit 1043 receives multiple pairs of the second design patch image and the second photographic patch image from the retrieval unit 1042 and stores them as a dictionary.

[0043] The image generation unit 1044 acquires a first design image from either the patch image acquisition unit 1041 or the retrieval unit 1042, and acquires a second photographic patch image from the retrieval unit 1042. The image generation unit 1044 assigns the second photographic patch image to the cropping positions of the first design patch image from the first design image. The image generation unit 1044 assigns the retrieved second patch image to each cropping position of the first patch image in the first design image, thereby generating a reference image.

[0044] Furthermore, regarding the configuration of sampling points in the patch image acquisition unit 1041, it is envisioned that all pixels of the first design image are cut as patch images at least once. Therefore, when the sampling points are configured densely, the number of times a portion of the image is cut as a patch image increases, resulting in an increase in the number of pasted second photographic patch images. Therefore, in the reference image generation unit 1044, the number of clues during reference image generation increases, thus improving the image quality of the reference image. On the other hand, when the sampling points are configured densely, the number of times the second photographic patch image is extracted by the retrieval unit 1042 (the number of queries) increases, thus increasing the computational load. Therefore, the number of sampling points should be set considering a trade-off between the image quality and the computational load of the reference image.

[0045] Furthermore, for areas of the image where no pattern exists in the first design image, the sampling points can be made sparser to reduce the number of sampling points. Additionally, if the extracted second design patch image is the same as the second design patch image already registered in the dictionary storage unit 1043, the transformation unit 104 can delete the second photographic patch image corresponding to the extracted second design patch image without registering a new one. Alternatively, it can generate an average image of the second photographic patch image corresponding to the extracted second design patch image and the already registered second photographic patch image, and register this average image instead of the already registered second photographic patch image.

[0046] Alternatively, the second design patch image and the corresponding multiple second photographic patch images can be stored as an image pair in the dictionary storage unit 1043. For example, the display unit 108 can display the second design patch image and the multiple second photographic patch images on the screen for the user to select.

[0047] When the cutting positions of the second design patch image are close together, the probability of images at the same position being repeatedly cut is high. Therefore, the dictionary storage unit 1043 is not limited to storing patch images, but may also store image pairs of the second design image and the second photographic image, and position information related to each cutting position on the second design image and the second photographic image. The position information may be, for example, the center coordinates of the cutting position and the patch size, or a set of coordinates of the upper left and lower right of the cut patch image. The retrieval unit 1042 cuts patch images from the second design image and the second photographic image based on the position information and performs image matching.

[0048] Alternatively, the vector data can be retained as vector data obtained by reducing the dimensions of at least one of the second design patch image and the second photographic patch image. For example, principal component analysis can be performed on at least one of the second design patch image and the second photographic patch image, retaining only the vector data of the patch image expressed by principal components with predetermined contribution rates. This reduces the required dictionary storage capacity.

[0049] Alternatively, conventional clustering methods can be used to cluster the second design patch image and the second photographic patch image stored in the dictionary storage unit 1043. For example, the K-means clustering method can be used. Alternatively, a hierarchical representation such as a tree structure can be used to classify them into many classes.

[0050] Next, refer to Figure 3 This is a conceptual diagram illustrating the use of a tree structure to store patch images in the dictionary storage section 1043.

[0051] Figure 3 This is a plotting graph that shows the second design patch images contained in the dictionary and categorizes them according to the patterns along the x-axis and y-axis. The closer the plots are, the more similar the second design patch images are. Figure 3 In the diagram, the second design patch image is clustered based on pattern similarity, and is segmented into multiple classes defined by dashed lines and coordinate axes. An example is shown where the classes are linked together using a tree structure to maintain the dictionary.

[0052] Determine representative points for each class. A representative point is an image representing a pattern of multiple second design patch images contained within the class. This can be either selected from the multiple second design patch images contained within the class, or a newly calculated image such as the average of multiple second design patch images. Treat each representative point as a node to form a tree structure.

[0053] Specifically, in Figure 3 In the example, the representative point 302-1 of classes a and b, and the representative point 302-2 of classes c, d, and e are formed as nodes at the same level. As subordinate levels of representative point 302-1, the representative points 303-1 of class a and 303-2 of class b are formed as nodes at the same level. Similarly, as subordinate levels of representative point 302-2, the representative point 303-3 of class c, and the representative point 303-4 of classes d and e are formed as nodes at the same level. Furthermore, as subordinate levels of representative point 303-4, the representative points 304-1 of class d and 304-2 of class e are formed as nodes at the same level.

[0054] Here, we will describe an example of how the retrieval unit 1042 retrieves a second design patch image that is similar to the first design patch image based on a tree structure.

[0055] The retrieval unit 1042 compares the first design patch image 350 with representative points of the nodes forming the tree structure, and sequentially searches for nodes with similar patterns in the lower sub-layers. Specifically, in Figure 3 In the example, the similarity between the first design patch image 350 and each of the two representative points 302-1 and 302-2 at the top layer of the tree structure is compared, for example, by calculating the distance between the images. Here, we assume that the representative point 302-2 is close to the first design patch image 350 (that is, similar). The retrieval unit 1042 calculates the distance between the first design patch image 350 and the lower layers (representative points 303-3 and 303-4) of the representative point 302-2. Here, we assume that the first design patch image 350 is close to the representative point 303-4, and similarly, we calculate the distance between the first design patch image 350 and the lower layers (representative points 304-1 and 304-2) of the representative point 303-4. Here, when the final design patch image 350 is close to the representative point 304-2, the retrieval unit 1042 only needs to calculate the distance between the second design patch image and the first design patch image 350 contained in class e of the representative point 304-2.

[0056] according to Figure 3The tree-like structure shown indicates that when the retrieval unit 1042 searches for a second design patch image similar to the first design patch image, it searches sequentially from the upper to the lower levels of the tree structure for second design patch images with similar patterns. This reduces the number of searches (distance calculations) compared to searching all second design patch images contained in the dictionary. In this regard, the more data stored in the dictionary, the more beneficial it is, as it can shorten the retrieval time and improve retrieval efficiency.

[0057] Furthermore, during the tree structure construction or update phases, only one pair of identical, or duplicate, node images is required. Therefore, multiple node images (the second design patch image and its corresponding second photographic patch image) with a similarity exceeding the threshold can be deleted. Additionally, node images can be averaged. For example, during the tree structure construction or update phases, the second design patch image serving as a node image may be identical, but the corresponding second photographic patch image is an actual photographed image, so the second photographic patch image may differ in brightness, etc. Therefore, by setting only one node image with a similarity exceeding the threshold, and averaging multiple second photographic patch images associated with the second design patch image serving as a node image, redundant node images can be reduced, thus reducing the data size of the dictionary.

[0058] In addition, Figure 3 In the example, a tree structure is envisioned based on the similarity of the pattern shape (bump, curvature such as whether it is a straight line or a curve, etc.) of the objects to be inspected contained in the second design patch image. However, a tree structure can also be constructed based on other similarity indicators such as the distribution of brightness values ​​and maintained as a dictionary.

[0059] In addition, Figure 3 In the example, the branches at the nodes of the tree structure used similarity to the representative points of the class, but it can be arbitrary as long as the rule can determine which class the first design patch image 350 belongs to. For example, a hyperplane that divides the region can also be used, in which case the same processing can be performed when the hyperplane is saved instead of the representative points at each node.

[0060] In addition, there are methods that improve retrieval efficiency without using a tree structure. For example, it is also possible to store... Figure 3 The representative points of the smallest region (303-1, 303-2, 303-3, 304-1, and 304-2) are used to calculate the distance between the first design patch image 350 and each representative point during retrieval, and to determine the class to which the first design patch image, which is used as the retrieval image, belongs.

[0061] By using the method described above in the dictionary storage unit 1043, the computation time spent searching for a second design patch image similar to the first design patch image can be shortened.

[0062] Furthermore, the dictionary storage unit 1043 can maintain different dictionaries depending on the type of image pairs acquired from the image storage unit 103. For example, different dictionaries can be maintained based on the minimum linewidth or minimum processing size determined during circuit design.

[0063] Furthermore, in this embodiment, it is envisioned that the dictionary storage unit 1043 is included in the transformation unit 104, but it is not limited thereto and may also exist inside the defect classification device 10. Alternatively, the dictionary storage unit 1043 may also exist outside the defect classification device 10. The transformation unit 104 may also access the externally located dictionary storage unit 1043 when generating reference images. Alternatively, the dictionary may also be stored in the image storage unit 103.

[0064] Furthermore, regarding the shape of each patch image, we assume a square, but it is not limited to this. It can be a trapezoid, parallelogram, or other quadrilaterals, or a triangle, pentagon, or other polygons. Alternatively, it can be any shape such as a circle or a cross. For example, the shape of the patch image can be appropriately determined based on the circuit pattern of the object being inspected, such as whether it has many straight lines or curves.

[0065] Furthermore, the size of the patch image is not limited to a fixed size; multiple sizes can be set. For example, the size of the patch image can be set according to the minimum line width or minimum processing size determined during circuit design, with smaller sizes for finer patterns.

[0066] Next, refer to Figure 4 as well as Figure 5 This section explains an example of pasting the image for the second photographic patch.

[0067] Figure 4 It shows four sampling points 41, four second photographic patch images 42 corresponding to each sampling point 41, and generated pixels 43 as reference images.

[0068] Each of the four sampling points 41 corresponds to one second photographic patch image 42. Therefore, the average of the pixel values ​​of each of the second photographic patch images 42 that overlap with the position of the generated pixel 43 is taken as the pixel value of the generated pixel 43. Alternatively, if the second photographic patch image 42 is associated with similarity, the weighted average of the pixel values ​​of each of the second photographic patch images 42 with that similarity as a weight can also be taken as the pixel value of the generated pixel 43.

[0069] in addition, Figure 5This refers to the case where one sampling point 41 corresponds to two second photographic patch images 42. In such cases... Figure 5 In the case where sampling point 41 is set to be repeated by two second photographic patch images 42 that are common but different in size, it can also be achieved by... Figure 4 The same method is used to calculate the pixel value of generated pixel 43. Furthermore, when pasting the large second photographic patch image 42, the partial image used as the reference image is considered to have a high similarity, so a weighted average, such as increasing the weight of the large second photographic patch image 42, can also be used as the pixel value of generated pixel 43.

[0070] Furthermore, when pasting the second photographic patch image, pasting can also be performed with sub-pixel precision. This is because there are still cases where the phases of the images in the first and second design patch images differ. Adjusting to a position slightly off-center from the location where the first design patch image was cut—that is, adjusting in sub-pixel units smaller than pixels—can sometimes provide better accuracy. Therefore, referring to the image generation unit 1044, the second design patch image can be aligned with the cutting position of the first design patch image with sub-pixel precision to determine the optimal pasting position.

[0071] Next, refer to Figure 6 The flowchart illustrates an example of the operation of the defect classification device 10 involved in the implementation method.

[0072] In step S601, the design image acquisition unit 101 acquires the first design image.

[0073] In step S602, the transformation unit 104 generates a reference image from the first design image using the above-described method.

[0074] In step S603, the calculation unit 105 calculates the confidence level of the reference image. The confidence level is calculated for each pixel of the reference image, for example, based on the similarity during the generation of the reference image. During the generation of the reference image, the similarity between the first design patch image and the second design patch image is calculated. The lower the similarity, the less likely it is that the expected data corresponding to the first design patch image does not exist in the dictionary stored in the dictionary storage unit 1043. Therefore, the pixel values ​​of the reference image generated from the second photographic patch image corresponding to the second design patch image with low similarity can be said to have low confidence.

[0075] On the other hand, the higher the similarity, the higher the confidence level of the pixel value of the reference image generated from the second photographic patch image corresponding to the second design patch image with higher similarity. Therefore, in the calculation of a pixel value in the reference image, when multiple second photographic patch images are used, the confidence level can be calculated based on statistics such as the average, variance, maximum or minimum value of the similarity among the multiple second photographic patch images, and combinations of two or more of them.

[0076] In step S604, the photographic image acquisition unit 102 acquires the first photographic image.

[0077] In step S605, the detection unit 106 compares the first photographic image and the reference image. Specifically, for each pixel at the same location, the difference between the pixel value of the first photographic image and the pixel value of the reference image is calculated. Alternatively, the difference can be calculated after smoothing at least one of the images, the first photographic image and the reference image, using an averaging filter or the like. The difference can also be calculated by sharpening at least one of the images, the first photographic image and the reference image, using an edge enhancement filter or the like.

[0078] Furthermore, the dynamic range can be adjusted for both the first photographic image and the reference image. For example, the difference caused by the difference in the average values ​​(DC component) between the two images can be eliminated by subtracting the average pixel value of the combined first photographic image and the reference image, and then the first photographic image and the reference image can be compared. Alternatively, the comparison can be performed after standardizing each image. Additionally, the difference can be represented using correlation values. For example, the entire image or a portion of the image can be cropped, and the standardized cross-correlation between the cropped portions can be calculated. The higher the standardized cross-correlation value, the smaller the difference between the images.

[0079] In step S606, the detection unit 106 determines whether there is a defect. Specifically, it determines whether the difference calculated in step S605 is above a threshold. If the difference is above the threshold, it is determined that there is a defect, and the process proceeds to step S607. If the difference is less than the threshold, it is determined that there is no defect, and the process ends.

[0080] In step S607, the detection unit 106 detects the defect location. Specifically, the detection unit 106 can infer the defect location from the pixel location that was determined to be defective in step S606.

[0081] In step S608, the classification unit 107 classifies the defects at the defect location and outputs information indicating the type of defect. For example, classification is performed using a recognizer. The recognizer includes a trained model, which is obtained by training a machine learning model using a first photographic image containing the defect, a reference image, the confidence level of the reference image, and the defect location as input data, and the defect type as correct data. The classification unit 107 inputs the first photographic image, the defect location, and the confidence level into the trained model and outputs the type of defect at that location. The machine learning model can utilize support vector machines, random forests, deep learning, or similar methods.

[0082] Furthermore, the identifier outputs the category with the highest probability among multiple defect categories as the classification result. Alternatively, it can output the probability of each of the multiple defect categories as the classification result. For example, for data where the maximum probability value is less than a predetermined threshold, the defect category can be omitted, and the probability value of each category can be displayed to the user.

[0083] Furthermore, the classification unit 107 is not limited to classifying defects using a recognizer that includes a learned model; it can also classify defects using a rule base based on the confidence level of the reference image.

[0084] In step S609, the display unit 108 indicates the type of the detected defect to the user. Furthermore, the defect location may also be indicated to the user. For example, the defect location in the first photographic image or a reference image can be overlaid on the image, or the confidence level at the defect location in the reference image can be displayed. The display unit 108 may also overlay the confidence level at a position corresponding to the defect location in the first photographic image. In the case of overlay display, the confidence level can be displayed by color-coding using a heatmap, or information such as the reference image ID that can identify the detected defect can be displayed. Additionally, if no defect is detected, the user can be notified that there is no abnormality (normal product).

[0085] Furthermore, during the learning process of the learned model used in the classification section 107, the first photographic image, the defect location, reference images with a confidence level of 105 or higher, and their corresponding confidence levels are used as input data, while the type of defect at the defect location is used as correct data to learn the machine learning model. Since inspection results based on reference images with low confidence levels may negatively impact the machine learning results, they are excluded from the learning data beforehand, thereby improving the accuracy of the machine learning.

[0086] Alternatively, reference images with low confidence can also be used as input data. In this case, learning can be performed in a way that emphasizes the confidence value, thereby reducing the weight of classification results based on reference images with low confidence and increasing the weight of classification results based on reference images with high confidence.

[0087] During the learning process of the learned model, input data and correct data are learned using pre-prepared data. During inference by the learned model, in classifying defects of the inspected object, if the defect is similar to the image pattern of a pre-prepared defect type, high classification accuracy (correctness) can be expected. However, if even the same type of defect produces a new image pattern, the (correctness) may decrease. Therefore, if the probability of the classification result output from the classification unit 107 is above a threshold, the first photographic image, reference image, and confidence score can be added as learning data to the learned model for application in the machine learning model's learning.

[0088] Furthermore, during the learning process of the already learned model, circuit shape information at the defect location can be added as input data. This circuit shape information includes, for example, the width of the wiring, the corner portions of the wiring, and the shape of the broken wire. When circuit shape information is used as input data during learning, it can also be input during inference of the already learned model at the defect location from the first photographic image.

[0089] Next, refer to Figure 7 This is the first example of the classification results for defects in classification section 107.

[0090] Figure 7 The input and output of the recognizer (learned model) included in the classification unit 107 are shown. Image 701, the confidence level 702 of image 701, and the first photographic image 703 are input to the recognizer 704. Figure 7 In the example, the confidence level 702 of the reference image 701 is set to 90%. In addition, it is conceivable that a defect 710 exists in the object being inspected, and the defect 710 is reflected in the first photographic image 703.

[0091] The recognizer 704 classifies defects based on inputs including reference image 701, confidence level 702, and the first photographic image 703, and outputs a result indicating the type of defect. As an example, it displays an image 705 indicating the location of defect 710 and defect information 706 related to the type of defect 710. The image 705 indicating the location of defect 710 is, for example, a difference image of reference image 701 and the first photographic image 703. The defect information 706 displays the type of defect 710 and confidence level 702. Specifically, in... Figure 7In the example, it displays "Type: Hole defect, Confidence level: 90%". Alternatively, confidence level 702 can be omitted from defect information 706.

[0092] Next, refer to Figure 8 This is the second example illustrating the classification results of defects in classification section 107.

[0093] Figure 8 Showing the target and Figure 7 The same recognizer 704 has the same input and output, but the confidence level 802 is different when referring to image 801. Figure 8 In this case, let's assume a relatively low confidence level of 30%.

[0094] The reference image 801, confidence level 802, and first photographic image 803 are input to the recognizer 704, and the recognizer 704 outputs an image 804 indicating the location of the defect and defect information 805.

[0095] In image 804, which indicates the location of the defect, the difference between reference image 801 and the first photographic image 803 is shown. The part that is interrupted in reference image 801 is presented as a defect in image 804. In the recognizer 704, considering that the confidence level 802 of reference image 801 is low, the possibility that the defect based on reference image 801 is inaccurate is high, and the defect information 805 displays "Type: False defect, Confidence level: 30%".

[0096] Furthermore, in the implementation, inspections are envisioned in circuit design, such as semiconductor design, but are not limited to this. For example, the first inspection object could also be a building, in which case the first design image is based on 3D design data related to the building. That is, the defect classification device 10 according to the implementation can also compare a pre-conceptual drawing (referencing image) of the building's appearance based on 3D design data generated using computer-aided design software with a photographic image of the building after its actual completion, and detect the presence or absence of defects based on the confidence level of the pre-conceptual drawing. For example, if the confidence level of the pre-conceptual drawing is above a threshold, the detection unit 106 considers it to be defective and detects the defect location, and the display unit 108 displays information related to the defect location. On the other hand, if the confidence level of the pre-conceptual drawing is below the threshold, the detection unit 106 determines that there is no defect. Furthermore, since the confidence level is below the threshold, the display unit 108 can also indicate to the user that it has not been determined to be a defect.

[0097] In addition to the design data that determines the structure of the first inspection object, a first design image can also be generated, for example, with additional information on the raw materials such as whether it is a wooden structure or reinforced concrete, and color-related information such as the color of the walls.

[0098] That is, as long as a product or building is manufactured based on data generated using computer-aided design software, the defect classification device 10 described in the implementation method can be applied.

[0099] Next, Figure 9 An example of the hardware structure of the defect classification device 10 according to the above embodiment is shown.

[0100] The defect classification device 10 includes a CPU (Central Processing Unit) 901, RAM (Random Access Memory) 902, ROM (Read Only Memory) 903, a storage device 904, a display device 905, an input device 906, and a communication device 907, which are connected by a bus.

[0101] CPU 901 is a processor that performs arithmetic and control processing according to a program. CPU 901 uses a predetermined area of ​​RAM 902 as its working area and, in cooperation with programs stored in ROM 903 and storage device 904, performs the various processes related to defect classification device 10.

[0102] RAM 902 is a type of memory such as SDRAM (Synchronous Dynamic Random Access Memory). RAM 902 functions as the operating area of ​​CPU 901. ROM 903 is a non-rewritable memory that stores programs and various information.

[0103] Storage device 904 is a means of writing and reading data into magnetic recording media such as HDDs, semiconductor-based storage media such as flash memory, or storage media capable of magnetic or optical recording such as HDDs (Hard Disc Drives). Storage device 904 performs data writing and reading into the storage medium under the control of CPU 901.

[0104] Display device 905 is a display device such as LCD (Liquid Crystal Display). Display device 905 displays various information based on display signals from CPU 901.

[0105] Input device 906 includes input devices such as a mouse and keyboard. Input device 906 accepts information input from user operations as indication signals and outputs the indication signals to CPU 901.

[0106] The communication device 907 communicates with external devices via a network under the control of the CPU 901.

[0107] According to the embodiments described above, when generating a reference image of a mask image of an inspection object as a defect-free object based on the first design image, a confidence level of the reference image is calculated for each pixel, and defects of the inspection object are classified according to the confidence level. Thus, for example, if the confidence level of the reference image is low, it is possible to determine that the defect detected using the reference image may be a false defect. Therefore, defect classification can be performed based on the reproducibility of the reference image, improving classification accuracy.

[0108] The instructions shown in the processing sequence described in the above embodiments can be executed based on a program that is software. By pre-storing and reading the program in a general-purpose computer system, the same effect as the control action of the defect classification device described above can be obtained. The instructions described in the above embodiments are recorded as programs that can be executed by a computer on a disk (flexible optical disc, hard disk, etc.), optical disc (CD-ROM, CD-R, CD-RW, DVD-ROM, DVD±R, DVD±RW, Blu-ray disc, etc.), semiconductor memory, or similar recording media. As long as the recording medium can be read by a computer or embedded system, its storage format can be arbitrary. As long as the computer reads the program from the recording medium and executes the instructions described in the program using the CPU, the same action as the control of the defect classification device described in the above embodiments can be achieved. Of course, in the case where the computer obtains or reads the program, it can also obtain or read it via a network.

[0109] Alternatively, a portion of the processing for implementing the implementation method may be executed by an OS (operating system), database management software, network, or other MW (middleware) running on the computer based on instructions from a program installed from a recording medium onto the computer or embedded system.

[0110] Furthermore, the recording medium in the implementation is not limited to media independent of a computer or embedded system, but also includes recording media that download and store or temporarily store programs transmitted via LAN, Internet, etc.

[0111] Furthermore, the recording medium is not limited to one; when the processing in the implementation is performed from multiple media, the recording medium in the implementation is also included, and the structure of the medium can be arbitrary.

[0112] Furthermore, the computer or embedded system in the embodiments is used to execute the various processes in the embodiments based on the program stored in the recording medium, and can also be any structure such as a device consisting of a personal computer, a microcomputer, or a system in which multiple devices are connected by a network.

[0113] In addition, the computer in the embodiments is not limited to personal computers, but also includes computing devices, microcomputers, etc. included in information processing equipment. It is a general term for devices and apparatuses that can use programs to implement the functions in the embodiments.

[0114] Several embodiments of the invention have been described, but these embodiments are given by way of example and are not intended to limit the scope of the invention. These embodiments can be implemented in various other ways, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention as well as in the invention as described in the claims and its equivalents.

Claims

1. A defect classification apparatus comprising: a first acquisition section that acquires a first design image relating to a first inspection object, the first design image being an image based on design data created by design software; a second acquisition section that acquires a first photographic image, the first photographic image being obtained by photographing the first inspection object based on the design data; a conversion section that converts the first design image into a reference image expressed using a second photographic image, the second photographic image being obtained by photographing a second inspection object that has no defects; a calculation section that calculates a confidence of the reference image based on a degree of similarity of the design data of the first inspection object and the second inspection object at the time of creating the reference image; a detection section that compares the reference image and the first photographic image to detect a defect in the first inspection object; and a classification section that classifies the defect based on the confidence, wherein a second photographic patch image that is a partial image of the second photographic image and a second design patch image that is a partial image of a second design image based on design data relating to the second inspection object are associated, the conversion section pastes a second photographic patch image associated with a second design patch image similar to a first design patch image that is a partial image cut out from the first design image, in correspondence with a cutout position of the first design patch image, to thereby create the reference image, the calculation section sets the confidence of the cutout position to be higher as the degree of similarity of the first design patch image and the second design patch image is higher, the classification of the defect includes a defect that should be corrected, a defect that does not become a problem in terms of operation of a circuit, and a false defect that is erroneously detected as a defect for a qualified product, and the classification section classifies the defect as a false defect when the confidence corresponding to the position of the defect is less than a threshold value.

2. The defect classification apparatus according to claim 1, wherein the degree of similarity is set to be higher as a difference between the first design patch image and the second design patch image is smaller.

3. The defect classification apparatus according to claim 1, wherein the degree of similarity is set to be higher as a value of a normalized cross-correlation of the first design patch image and the second design patch image is larger.

4. The defect classification apparatus according to any one of claims 1 to 3, wherein the calculation section calculates a statistic amount relating to the degree of similarity of the first design patch image and each of a plurality of second design patch images as the confidence of a pixel position at which the plurality of second photographic patch images overlap.

5. The defect classification apparatus according to any one of claims 1 to 3, wherein the detection section compares the reference image and the first photographic image, detects a region of a pixel having a difference of a threshold value or more as the defect, and acquires the position of the defect and the confidence of the reference image corresponding to the position of the defect.

6. The defect classification apparatus according to claim 5, wherein ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ The detection section detects the position of the defect based on pixel values of images obtained by normalizing the reference image and the first captured image, respectively.

7. The defect classification apparatus according to any one of claims 1 to 3, wherein The classification section inputs the first captured image and the confidence corresponding to the position of the defect to a learned model that has learned in a manner of inputting an image having a defect and a corresponding confidence and outputting a kind of defect, thereby classifying the kind of the defect related to the first captured image.

8. The defect classification apparatus according to any one of claims 1 to 3, wherein The defect classification apparatus further includes a display section that displays the position of the defect and the kind of the defect in the first captured image.

9. A defect classification method, wherein a first design image related to a first inspection object is acquired, the first design image being an image based on design data made by using design software, a first captured image is acquired, the first captured image being obtained by capturing the first inspection object generated based on the design data, the first design image is transformed into a reference image expressed using a second captured image, the second captured image being obtained by capturing a second inspection object having no defect, a confidence of the reference image is calculated according to similarity of design data of the first inspection object and the second inspection object at the time of generating the reference image, the reference image and the first captured image are compared, and a defect in the first inspection object is detected, and the defect is classified based on the confidence, wherein a second captured patch image that is a partial image of the second captured image and a second design patch image that is a partial image of a second design image based on design data related to the second inspection object are associated, a second captured patch image associated with a second design patch image similar to a first design patch image that is a partial image cut out from the first design image is pasted in correspondence with a cut-out position of the first design patch image, thereby generating the reference image, the higher the similarity of the first design patch image and the second design patch image, the higher the confidence of the cut-out position is set, the classification of the defect includes a defect to be corrected, a defect that does not become a problem in terms of operation of a circuit, and a false defect that is erroneously detected as a defect for a qualified product, in a case where the confidence corresponding to the position of the defect is less than a threshold value, the defect is classified as a false defect.

10. An information recording medium storing a defect classification program for causing a computer to function as: a first acquisition unit that acquires a first design image related to a first inspection object, the first design image being an image based on design data made by using design software; a second acquisition unit that acquires a first captured image, the first captured image being obtained by capturing the first inspection object generated based on the design data; a conversion unit that converts the first design image into a reference image expressed using a second photographic image obtained by photographing a second inspection target that has no defect; a calculation unit that calculates a confidence level of the reference image based on similarity of design data of the first inspection target and the second inspection target at the time of generating the reference image; a detection unit that compares the reference image and the first photographic image to detect a defect in the first inspection target; and a classification unit that classifies the defect based on the confidence level, wherein a second photographic patch image that is a partial image of the second photographic image and a second design patch image that is a partial image of a second design image based on design data related to the second inspection target are associated, the conversion unit pastes a second photographic patch image associated with a second design patch image similar to a first design patch image that is a partial image obtained by cutting a portion from the first design image, at a position corresponding to a cut position of the first design patch image, thereby generating the reference image, the higher the similarity of the first design patch image and the second design patch image, the higher the confidence level of the cut position set by the calculation unit, the classification of the defect includes a defect that should be corrected, a defect that does not become a problem in terms of operation of a circuit, and a false defect that is erroneously detected as a defect for a qualified product, the classification unit classifies the defect as a false defect when the confidence level corresponding to the position of the defect is less than a threshold value. ​

Images