Inspection aid
By optimizing the extraction and storage of image data through inspection auxiliary devices, the problem of insufficient image data for specific classification categories was solved, improving the learning efficiency and classification accuracy of machine learning models and ensuring the accuracy of defect detection.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SCREEN HOLDINGS CO LTD
- Filing Date
- 2025-12-31
- Publication Date
- 2026-07-14
Smart Images
Figure CN122391054A_ABST
Abstract
Description
Technical Field
[0001] The subject matter disclosed in this specification relates to an inspection aid. Background Technology
[0002] Traditionally, in the inspection of printed circuit boards (PCBs), inspection equipment is used to examine images of various areas of the PCB and extract candidate defect images that are deemed potentially defective. Then, a defect confirmation operation (so-called verification operation) is performed where operators visually confirm the areas on the PCB corresponding to the candidate defect images. The candidate defects extracted by the inspection equipment include genuine defects with a high probability of becoming quality problems and false reports that have no real possibility of becoming quality problems. In the aforementioned defect confirmation operation, operators visually identify genuine defects from the candidate defects extracted by the inspection equipment.
[0003] Furthermore, Patent Document 1 proposes an appearance inspection system comprising: a primary inspection unit that determines defects based on images of the object without using machine learning; and a secondary inspection unit that uses a machine learning model to distinguish between genuine defects and over-defective products (false positives). Thus, only genuine defects are detected during appearance inspection, preventing over-defective products from being detected as defects, thereby improving productivity.
[0004] To use the machine learning model described above for defect detection, training data for the machine learning model needs to be prepared in advance. When preparing this training data, a plurality of images (e.g., more than ten thousand) containing defect candidates are displayed on a monitor in a predetermined number. The operator visually examines each displayed image and determines whether the defect candidate in each image is a real defect or a false alarm. If the defect candidate is a real defect, the operator further determines the type of the real defect. The operator inputs these determination results using a mouse or keyboard and performs teaching operations corresponding to each image. This is how the training data described above is prepared (Patent Document 2).
[0005] Patent Document 1: Japanese Patent Application Publication No. 2021-177154
[0006] Patent Document 2: Japanese Patent Application Publication No. 2021-139769
[0007] However, when the training dataset has low comprehensiveness or insufficient image data for a specific classification category, it can lead to reduced learning efficiency and deterioration in the classification accuracy of the learned model. For example, when the incidence of a particular defect type is relatively low, the amount of image data will be limited. In such cases, it may be difficult to fully learn the features of defect types with limited data. Furthermore, there is a risk of overfitting the training data for defect types with limited image data. Summary of the Invention
[0008] The purpose of this invention is to provide a technique that can suppress insufficient data quantity in image data of a specific classification category.
[0009] To address the aforementioned issues, a first-mode inspection assistance device includes: a data storage unit that stores image data associated with any one of a plurality of inspection result information; and a data extraction unit that extracts image data for machine learning from the plurality of image data stored in the data storage unit, wherein the plurality of inspection result information indicates the presence or absence of defects or the type of defects, and the data extraction unit extracts a predetermined number of image data for each of the plurality of inspection result information.
[0010] In the second method, in the inspection auxiliary device of the first method, the inspection result information includes attribute information, which indicates the presence or absence of defects or the type of defects detected based on the inspection logic.
[0011] In the third method, in the inspection auxiliary device of the second method, the inspection result information includes verification information, which indicates the presence or absence of defects or the type of defects based on the operator's evaluation.
[0012] In a fourth method, in a third-party inspection auxiliary device, a plurality of inspection result information includes first false alarm information where the attribute information indicates the existence of a first defect and the verification information indicates the absence of the first defect, and the data extraction unit extracts image data associated with the first false alarm information.
[0013] In the fifth method, in the inspection auxiliary device of the fourth method, the inspection result information further includes first true defect information where the attribute information shows the existence of the first defect and the verification information shows the existence of the first defect, and the data extraction part extracts the image data associated with the first false information and the image data associated with the first true defect information respectively.
[0014] In the sixth method, in the inspection auxiliary device of the fifth method, the data extraction unit sets the difference between the number of image data extracted associated with the first false information and the number of image data extracted associated with the first true defect information to be below a predetermined threshold.
[0015] In the seventh method, in any of the fourth to sixth methods of the inspection auxiliary device, the inspection result information further includes second false alarm information where the attribute information indicates the existence of a second defect and the verification information indicates the absence of the second defect, and the data extraction part extracts the image data associated with the first false alarm information and the image data associated with the second false alarm information respectively.
[0016] In the eighth method, in any of the first to seventh methods of the inspection assistance device, the data extraction unit sets the difference in the number of image data extracted for at least two types of inspection result information among a plurality of inspection result information to a predetermined threshold or less.
[0017] In the ninth method, in any of the fourth to eighth methods of the inspection auxiliary device, the inspection result information includes foreign object false alarm information in which the attribute information indicates the presence of a foreign object defect and the verification information indicates the absence of the foreign object defect, and the data extraction unit extracts the image data associated with the foreign object false alarm information.
[0018] In the tenth embodiment, in any of the first to ninth embodiments of the inspection aid device, the data extraction unit extracts more image data associated with inspection result information of severe defects compared to image data associated with inspection result information of minor defects.
[0019] In the eleventh method, in any of the first to tenth methods of the inspection assistance device, the data extraction unit preferentially extracts image data with low similarity to other portraits from a plurality of image data associated with the same type of inspection result information.
[0020] According to the inspection assistance device of the first to eleventh methods, it is possible to extract a necessary and sufficient amount of image data for each inspection result. Therefore, it is possible to suppress the insufficient amount of image data for a specific classification category.
[0021] According to the inspection auxiliary device of the second method, a specified number of image data can be extracted for each verification information.
[0022] Based on the third-party inspection assistance device, a specified number of image data can be extracted for each combination of attribute information and verification information.
[0023] According to the fourth inspection aid device, it is possible to ensure that image data used for learning about false reports of the first defect is protected.
[0024] According to the inspection aid device of the fifth method, it is possible to ensure, both necessary and sufficient, the image data used for learning about false and true defects of the first defect.
[0025] According to the inspection aid device of the sixth method, the bias between false reports and true defects regarding the first defect in the learning dataset can be reduced.
[0026] According to the seventh inspection aid device, it is possible to ensure, both necessary and sufficient, the false image data used for learning different types of defects.
[0027] According to the inspection assistance device of the eighth method, it is possible to reduce the bias of image data associated with at least two types of inspection result information in the learning dataset.
[0028] According to the inspection aid device of the ninth method, it is possible to ensure, both necessary and sufficient, that image data of removable foreign object defects are used as learning data.
[0029] According to the inspection aid device of the tenth method, it is possible to prioritize the use of image data of serious defects as learning data.
[0030] According to the inspection assistance device of method eleven, by prioritizing the selection of image data with low similarity, it is possible to construct a learning dataset with a wider range of features. Attached Figure Description
[0031] Figure 1 This is a diagram illustrating a defect inspection system that includes an inspection aid device for implementing the embodiments.
[0032] Figure 2 It is a graph that represents the multiple types of defects that image data can contain.
[0033] Figure 3 It is a diagram that conceptually represents the data stored in the data storage unit.
[0034] Figure 4 It means Figure 1 The diagram shown is a block diagram of the hardware structure of the inspection auxiliary device.
[0035] Explanation of reference numerals in the attached figures: 1: Inspect auxiliary devices 11: Data Storage Department 13: Data Extraction Department Detailed Implementation
[0036] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. Furthermore, in the drawings, the dimensions and quantities of the various parts will be exaggerated or simplified for ease of understanding.
[0037] <1. Example>
[0038] Figure 1 This is a diagram illustrating a defect inspection system 100 including the inspection aid 1 of the embodiments. The defect inspection system 100 is, for example, a system for performing visual inspection of a printed wiring substrate 9 (hereinafter also simply referred to as "substrate 9") before electronic components are mounted. A pattern (e.g., a wiring pattern or electrode pattern formed of copper) is formed on the surface of the substrate 9, which is the object of this visual inspection.
[0039] The inspection auxiliary device 1 has the function of generating a learned model for defect detection of the substrate 9 and the function of preparing learning data for generating the learned model. In addition to the inspection auxiliary device 1, the defect inspection system 100 also has a primary inspection device 2 and a secondary inspection device 3.
[0040] The primary inspection device 2 is a device that performs a single inspection on the substrate 9 to detect defects and sends image data containing the defects to the data storage unit 11 of the inspection auxiliary device 1. The primary inspection device 2 includes: an imaging unit 21, including an image sensor for capturing images of the substrate 9; and a computer 23, including a processor such as a CPU (Central Processing Unit) and memory such as RAM (Random Access Memory). The primary inspection device 2 uses the imaging unit 21 to capture images of the substrate 9 and acquire images. These images are, for example, grayscale images of the entire area on the substrate 9 that is the object of inspection. Furthermore, the substrate image acquired using the imaging unit 21 can also be, for example, a multi-grayscale color image.
[0041] The primary inspection device 2 detects defects based on several inspection logics. The inspection logic is an image processing method used to detect defects. The primary inspection device 2 is a device that detects defects according to these inspection logics. Therefore, the defects detected by the primary inspection device 2 are not limited to true defects; sometimes they are false alarms.
[0042] As an example of inspection logic, a comparison check can be cited. A comparison check compares the main image with the inspection image, and identifies a defect if the difference in pixel values between the main image and the inspection image exceeds a threshold. For example, the main image might be an image based on design data from a circuit pattern.
[0043] Another inspection logic can be called design rule inspection. Design rule inspection involves measuring the line width of patterns, the space between patterns, the size of pinholes, etc., within an inspection image, and identifying defects if the measured values do not meet the pre-set conditions.
[0044] The primary inspection device 2 performs inspections based on this inspection logic, thereby generating attribute information indicating the presence or absence of defects or the type of defects for each image data. This attribute information is then associated with the image data and stored in the data storage unit 11 of the inspection auxiliary device 1. Furthermore, the attribute information may also include the inspection logic used for the inspection and its inspection results. Additionally, the attribute information may include the position coordinates on the image where defects were detected. The primary inspection device 2 can also send the image data containing the attribute information to the secondary inspection device 3.
[0045] Figure 2 It is a graph representing the multiple defect types that image data 60 can contain. Figure 2 The image data 60 shown can contain three pattern elements 61, 62, and 63, indicated by the shading of parallel lines. In the image data 60, pattern elements 61-63 are bright areas with a brightness higher than the surrounding background area 64. The background area 64 is a dark area with a lower brightness than pattern elements 61-63.
[0046] Examples of defects detected by the inspection logic (e.g., design rule check) of the inspection device 2 include open circuit defects D11, gaps D12, and holes D13. An open circuit defect D11 is a broken line defect where a linear pattern element 61 is completely interrupted in the long side direction. A gap D12 is a defect where a portion of a linear pattern element 62 is missing in the width direction. A hole D13 is a defect where a portion of a planar pattern element 63 is missing in a dotted or similar manner. These defects are caused by missing pattern elements.
[0047] Furthermore, examples of defects detected by the inspection logic of the single-inspection device 2 include short-circuit defects D21, protrusions D22, and foreign objects D23. Short-circuit defect D21 is a defect caused by a short circuit resulting from the connection of pattern elements 61 and 62 that should not be connected. Protrusion D22 is a defect in which an unwanted protrusion in the pattern element 61 protrudes in the width direction. Foreign object D23 is a defect in which an unwanted element, such as an isolated dot, exists in the background area 64.
[0048] return Figure 1The secondary inspection device 3 is used for verification operations. These verification operations involve an operator visually confirming the defects detected by the primary inspection device 2 to determine whether the detected defects (defect candidates) are genuine defects or false alarms. The secondary inspection device 3 includes an imaging unit 31 and a computer 33. The computer 33 includes a processor, memory, a display, and input devices (keyboard and mouse, etc.).
[0049] The secondary inspection device 3 uses the imaging unit 31 to capture an image of the area of the substrate 9 where defects were detected by the primary inspection device 2. This image is, for example, a color image. An operator visually reviews the image displayed on the monitor of the computer 33, evaluates whether the defect indicated by the attribute information is a genuine defect or a false alarm, and inputs the evaluation result into the computer 33. The computer 33 sends the input evaluation result as verification information to the data storage unit 11 of the inspection auxiliary device 1. The verification information is associated with the image data sent to the data storage unit 11 by the primary inspection device 2.
[0050] Figure 3 This is a conceptual diagram representing the data stored in the data storage unit 11. Figure 3 In the example shown, the data stores an ID representing the image data, attribute information associated with that ID, and verification information. Here, the attribute information indicates the presence or type of detected defects. Additionally, the verification information indicates whether the defect is genuine or a false alarm. The attribute information and verification information are an example of inspection result information. The inspection result information in this example is a combination of attribute information indicating the presence or type of defects (defect candidates) and verification information indicating whether a defect is genuine or a false alarm. Inspection result information indicates the presence or type of defects.
[0051] Furthermore, a true defect indicated by the verification information means that the defect indicated by the attribute information is also true. That is, when verification information and attribute information are associated together with image data, the verification information is also equivalent to information indicating the type of defect. In addition, the verification information may also directly contain information indicating a specific type of defect.
[0052] In the following explanation, inspection result information that indicates the presence of a defect of a specific type in attribute information and that the verification information indicates a false alarm (absence of the first defect) is called "false alarm information." Indicating the presence of a defect refers to inputting information other than the absence of a defect. Specifically, inspection result information that indicates the presence of a foreign object defect in attribute information and that the verification information indicates a false alarm (absence of the foreign object defect) is called "foreign object false alarm information." Furthermore, inspection result information that indicates the presence of a defect of a specific type in attribute information and that the verification information indicates a true defect is called "true defect information."
[0053] Furthermore, the classification items for defect types used in the primary inspection device 2 and the classification items for defect types used in the secondary inspection device 3 may not always correspond one-to-one. For example, when an operator uses the secondary inspection device 3 for verification, there may be cases where a more detailed classification item is used than the one used in the primary inspection device 2. In such cases, even if both attribute information and verification information indicate the presence of a defect, the defect type indicated by the attribute information and the defect type indicated by the verification information may differ. In this invention, these are also treated as genuine defect information.
[0054] Furthermore, all image data stored in the data storage unit 11 does not necessarily need to have verification information. For example, such as Figure 3 As shown, for image data that is not detected by the primary inspection device 2, since it is not inspected by the secondary inspection device 3, verification information may not be assigned.
[0055] Figure 4 It means Figure 1 The diagram shows a block diagram of the hardware structure of the inspection auxiliary device 1. (As shown...) Figure 4 As shown, the inspection auxiliary device 1 is composed of a computer having a processor 41 such as a CPU, a memory 42 such as RAM, and a storage unit 43 such as an HDD (Hard-Disk Drive). The processor 41, memory 42, and storage unit 43 are electrically connected via a bus 44. In addition, a display 45 and an input device 46 are connected to the bus 44.
[0056] Figure 1 The data storage unit 11 shown is implemented using the storage unit 43. Furthermore, the data storage unit 11 can be implemented using a device such as a server connected to the bus 44 via a network. The storage unit 43 stores a program P. The program P is read from a non-transitory recording medium via a reading device and stored in the storage unit 43. Alternatively, the program P can also be stored in the storage unit 43 via a network such as the Internet.
[0057] Processor 41 performs arithmetic operations using memory 42 according to program P, such as... Figure 1 As shown, it functions as the data extraction unit 13, the labeling unit 15, and the learning model generation unit 17.
[0058] The data extraction unit 13 extracts image data for machine learning from the image data stored in the data storage unit 11 according to a predetermined extraction rule R1. The extraction rule R1 specifies the amount of image data to be extracted (i.e., the necessary and sufficient extraction amount) for each of the plurality of inspection result information. Furthermore, the extraction amount can be specified by an absolute number or by a relative number (e.g., a ratio). Additionally, the extraction rule R1 is preferably configured to be set by the user of the inspection assistance device 1.
[0059] The labeling unit 15 performs labeling processing on the image data extracted by the data extraction unit 13, wherein the label represents a specific classification category ( Figure 1 (Categories A, B, and C are shown). Furthermore, label assignment can be based on user input or performed automatically. In the case of user input, the labeling unit 15 can display image data on the display 45, and the user, after visually verifying the image data, can specify the assigned label. Alternatively, in the case of automatic label assignment, the labeling unit 15 can also assign labels to the image data corresponding to the category information associated with the image data. The learning model generation unit 17 generates a learned model by performing machine learning on the image data labeled by the labeling unit 15 as learning data. As the learning model, any machine learning model, including multilayer perceptron neural networks, support vector machines, discriminant functions, or Bayesian networks, can be used.
[0060] The learned model obtained by the learning model generation unit 17 can be, for example, a model that outputs verification information indicating whether a defect detected by the primary inspection device 2 is a genuine defect or a false alarm. In this case, the learned model can be used to automate the generation of verification information performed by the operator using the secondary inspection device 3.
[0061] <About the extraction rules>
[0062] Next, a specific example of extraction rule R1 will be explained. Furthermore, in the following explanation, the three different defect types will be referred to as the first defect, the second defect, and the third defect, respectively. The first defect, the second defect, and the third defect are equivalent to... Figure 2 Any one of the following defects shown: open circuit defect D11, notch D12, hole D13, short circuit defect D21, protrusion D22, foreign object D23, etc.
[0063] Extraction rule R1, for example, includes a rule that minimizes the difference in the number of image data extracted for at least two types of inspection result information from a plurality of inspection result information (i.e., sets the difference in the number of extractions below a specified threshold). In this case, it is possible to reduce the bias between image data associated with at least two types of inspection result information in the training dataset. For example, by making the number of image data extracted associated with true defect information (first true defect information, second true defect information, and third true defect information) related to the first, second, and third defects equal, it is possible to prepare a training dataset with unbiased data quantity for the first, second, and third defects.
[0064] Extraction rule R1 includes rules for extracting image data associated with false information (first false information) of the first defect and image data associated with true defect information (first true defect information) of the first defect. Alternatively, extraction rule R1 may include rules to minimize the difference in the number of these extracted image data (i.e., setting the difference in the number of extracted image data below a predetermined threshold). In this case, it is necessary and sufficient to ensure the image data used for learning about false and true defects of the first defect, and it is possible to reduce the deviation in the number of image data used for learning about false and true defects of the first defect in the learning dataset. Furthermore, extraction rule R1 may also include rules, more preferably for all types of defects, setting the difference between the number of extracted image data associated with false information (first false information) and the number of extracted image data associated with true defect information (first true defect information) below a predetermined threshold.
[0065] Extraction rule R1 may also include rules for extracting image data associated with false alarm information of the first defect (first false alarm information) and image data associated with false alarm information of the second defect (second false alarm information). Additionally, extraction rule R1 may include rules for minimizing the difference in the number of extracted image data (i.e., setting the difference in the number of extracted image data below a predetermined threshold). This ensures that the image data used for learning false alarms about different types of defects is adequately and sufficiently guaranteed, and reduces the bias in the number of false alarm image data about different types of defects in the learning dataset.
[0066] The extraction rule R1 may also include rules for extracting image data associated with false foreign object reports. For example, there is a possibility that a foreign object attached during inspection by the primary inspection device 2 may have been removed during evaluation by the secondary inspection device 3. Therefore, by extracting image data associated with false foreign object reports, it is possible to ensure, both necessary and sufficient, that image data of removable foreign object defects are used as learning data.
[0067] Extraction rule R1 can also include rules that prioritize extracting image data associated with inspection results indicating severe defects over image data associated with inspection results indicating minor defects. This ensures that image data with severe defects is prioritized for training purposes. Severe defects refer to, for example, defects that significantly impact the operation of the circuit.
[0068] Extraction rule R1 may also include a rule that prioritizes extracting image data with low similarity to other images from a plurality of image data associated with the same type of inspection result information. According to this rule, for example, in the case of extracting one hundred images from one thousand image data associated with one type of inspection result information, images with low similarity to each other can be extracted. Furthermore, the method for calculating the similarity between images is not limited to comparison of pixel values or comparison based on feature quantities.
[0069] As described above, according to the inspection assistance device 1 of this embodiment, for each of the plurality of inspection result information extracted by the data extraction unit 13, the necessary and sufficient extraction quantity can be determined using extraction rule R1. Therefore, it is possible to suppress the decrease in learning efficiency and the decrease in classification accuracy of the learned model caused by insufficient data quantity of image data for a specific classification category.
[0070] <2. Variations>
[0071] The implementation methods have been described above, but the present invention is not limited to the above content and can be modified in various ways.
[0072] The inspection result information does not necessarily have to include both attribute information and verification information; it may only include one of them.
[0073] Furthermore, substrate 9 is not necessarily limited to printed wiring substrate. Substrate 9 can also be, for example, a semiconductor substrate, a semiconductor packaging substrate, a glass substrate for flat panel display devices such as liquid crystal display devices and plasma display devices, a glass substrate for photomasks, a substrate for solar cell panels, etc.
[0074] While the invention has been described in detail, the foregoing description is exemplary in all respects and the invention is not limited thereto. It is understood that many modifications not illustrated can be conceived without departing from the scope of the invention. The structures described in the above embodiments and modifications can be appropriately combined or omitted as long as they do not contradict each other.
Claims
1. An inspection auxiliary device, wherein, have: The data storage unit stores image data associated with any one of the multiple types of inspection result information; and The data extraction unit extracts image data for machine learning from a plurality of image data stored in the data storage unit. The various types of inspection result information indicate the presence or absence of defects or the type of defects. The data extraction unit extracts a predetermined amount of image data for each of the plurality of inspection result information.
2. The inspection auxiliary device according to claim 1, wherein, The inspection result information includes attribute information, which indicates the presence or absence of defects or the type of defects detected based on the inspection logic.
3. The inspection auxiliary device according to claim 2, wherein, The inspection result information includes verification information, which indicates the presence or type of defects based on the operator's evaluation.
4. The inspection auxiliary device according to claim 3, wherein, The plurality of inspection result information includes a first false alarm where the attribute information indicates the presence of a first defect and the verification information indicates the absence of the first defect. The data extraction unit extracts the image data associated with the first false information.
5. The inspection auxiliary device according to claim 4, wherein, The inspection result information also includes the attribute information indicating the existence of the first defect and the verification information indicating the existence of a first true defect. The data extraction section extracts the image data associated with the first false information and the image data associated with the first true defect information, respectively.
6. The inspection auxiliary device according to claim 5, wherein, The data extraction unit sets the difference between the number of image data extracted associated with the first false information and the number of image data extracted associated with the first true defect information to be below a predetermined threshold.
7. The inspection auxiliary device according to any one of claims 4 to 6, wherein, The inspection result information also includes second false alarm information, where the attribute information indicates the existence of a second defect and the verification information indicates the absence of the second defect. The data extraction section extracts the image data associated with the first false information and the image data associated with the second false information, respectively.
8. The inspection auxiliary device according to any one of claims 1 to 7, wherein, The data extraction unit sets the difference in the number of image data extracted for at least two types of inspection result information from a plurality of inspection result information to below a predetermined threshold.
9. The inspection auxiliary device according to any one of claims 4 to 8, wherein, The inspection result information includes attribute information indicating the presence of foreign object defects and verification information indicating the absence of such defects, which constitutes a false foreign object report. The data extraction unit extracts the image data associated with the false foreign object report information.
10. The inspection auxiliary device according to any one of claims 1 to 9, wherein, Compared to image data associated with inspection results indicating minor defects, the data extraction unit extracts more image data associated with inspection results indicating severe defects.
11. The inspection auxiliary device according to any one of claims 1 to 10, wherein, The data extraction unit prioritizes extracting image data with low similarity to other portraits from a plurality of image data associated with the same type of inspection result information.