Inspection support device, inspection support method, and recording medium on which program is recorded

Abstract

The invention provides an inspection assistance device, an inspection assistance method, and a recording medium on which a program is recorded. The inspection assisting device assists inspection of a substrate having a pattern on the surface. A storage unit (301) stores a plurality of defect candidate images detected by a single inspection on a substrate image. An image classification unit (302) classifies the plurality of defect candidate images into a plurality of classified image groups on the basis of defect categories or fatality of defect candidates included in each of the defect candidate images by performing rule-based classification on the plurality of defect candidate images. An image classification unit (302) generates classification display data capable of displaying the plurality of defect candidate images on the display unit for each classification image group. As a result, it is possible to improve the accuracy of visual determination of the defect candidate image by an operator.

Description

Technical Field

[0001] The present invention relates to a technology for assisting inspection of a substrate having a pattern on its surface.

[0002] [Cross reference]

[0003] This application claims the benefit of priority from Japanese patent application JP 2023-118629 filed on July 20, 2023, and all disclosed contents of that application are incorporated into this application. Background Art

[0004] In the past, in the inspection of printed wiring boards, images of various regions of the printed wiring boards were inspected by an inspection device, and defect candidate images that were judged to be defective were extracted. Then, a defect confirmation operation (so-called verification operation) was performed in which an operator visually confirmed the region on the printed wiring board corresponding to the defect candidate image. The defect candidates extracted by the inspection device include true defects that are highly likely to become quality problems, and false reports that are substantially unlikely to become quality problems. In the above-mentioned defect confirmation operation, the operator visually extracts true defects from the defect candidates extracted by the inspection device.

[0005] In addition, in Japanese Patent Publication No. 2021-177154 (Document 1), a visual inspection system is proposed, which includes: a primary inspection unit, which performs non-conforming judgment based on an image obtained by photographing an object without using machine learning; and a secondary inspection unit, which uses a machine learning model to distinguish between true non-conforming products (true defects) and over-judged products (false reports). Thus, only true non-conforming products are detected in the visual inspection, and the situation where over-judged products are detected as non-conforming products is suppressed, thereby achieving an improvement in productivity.

[0006] In order to use the above-mentioned machine learning model for defect detection, it is necessary to prepare learning data for the machine learning model in advance. When preparing the learning data, a predetermined number of multiple (for example, more than 10,000) images containing defect candidates are displayed on the display, and the operator visually determines whether the defect candidate in each image is a true defect or a false report by observing each displayed image. In the case where the defect candidate is a true defect, the operator also determines the type of the true defect. The operator uses the mouse and keyboard to input these judgment results and perform teaching operations (i.e., annotations) corresponding to each image. In this way, the above-mentioned learning data is generated (refer to Japanese Patent Gazette No. 2021-139769 (Document 2)).

[0007] However, in order to create a machine learning model with high defect detection accuracy, it is necessary to suppress the deviation of teaching in the creation of the above-mentioned learning data. Specifically, in the above-mentioned teaching operation, the same judgment result needs to be taught to similar defect candidate images. However, in the creation of learning data, as mentioned above, teaching operations for multiple defect candidate images are required, so as time passes, the operator's judgment criteria change, and the judgment may deviate. As a result, in the machine learning model created using the learning data, the accuracy of defect detection becomes low, and re-teaching operations for multiple defect candidate images may also be required. Summary of the invention

[0008] The present invention is directed to an inspection assisting device, an inspection assisting method, and a recording medium having a program recorded thereon, and aims to improve the accuracy of visual judgment of defect candidate images by an operator.

[0009] A first embodiment of the present invention is an inspection assisting device that assists in the inspection of a substrate having a pattern on its surface, the inspection assisting device comprising: a storage unit that stores a plurality of defect candidate images detected through a single inspection of the substrate image; and an image classification unit that classifies the plurality of defect candidate images based on rules, classifies the plurality of defect candidate images into a plurality of classified image groups based on the defect type or lethality of the defect candidates contained in each defect candidate image, and generates classified display data capable of displaying the plurality of defect candidate images in a display unit according to each classified image group.

[0010] According to the present invention, it is possible to improve the accuracy of visual determination of defect candidate images by an operator.

[0011] A second embodiment of the present invention is an inspection assisting device according to the first embodiment, further comprising: a display unit, which displays the plurality of defect candidate images according to each of the classified image groups; an input receiving unit, which receives an input of an evaluation of defects or false reports made by an operator who visually observes each defect candidate image displayed on the display unit as an evaluation of the authenticity of each defect candidate image; a learning data set generating unit, which generates a learning data set by corresponding the authenticity evaluation with each defect candidate image in the classified display data; and a learning unit, which performs machine learning on an initial model using the learning data set, thereby producing a learned model for AI inspection of substrate images.

[0012] A third aspect of the present invention is the inspection auxiliary device of the second aspect, further comprising a temporary setting unit, which sets a temporary authenticity evaluation for each defect candidate image included in each classified image group based on the defect category or fatality corresponding to each classified image group. When the input receiving unit receives an input different from the temporary authenticity evaluation, the learning data set generating unit associates the authenticity evaluation received by the input receiving unit with the defect candidate image, and associates the temporary authenticity evaluation with the defect candidate image in other cases.

[0013] In the fourth mode of the present invention, in the inspection auxiliary device of any one of the first to third modes, in the image classification unit, each defect candidate image is classified based on the lethality of the defect candidate contained in each defect candidate image. The lethality of the defect candidate is determined based on the size of the defect candidate contained in each defect candidate image and the width of the pattern element or background area at the position of the defect candidate.

[0014] According to a fifth aspect of the present invention, in the inspection assisting device of any one of the first to third aspects (or any one of the first to fourth aspects), the image classification unit classifies each defect candidate image based on the lethality of the defect candidate contained in each defect candidate image. The lethality of the defect candidate is determined based on the concentration of the defect candidate contained in each defect candidate image.

[0015] In a sixth aspect of the present invention, in the inspection assisting device of any one of aspects 1 to 3 (or any one of aspects 1 to 5), in the image classification unit, each defect candidate image is classified based on the defect category of the defect candidate contained in each defect candidate image. The image classification unit transforms each defect candidate image into a first image represented by a first color that is any one of RGB. The image classification unit generates a second image that represents a reference image that does not have the defect candidate contained in each defect candidate image by a second color in RGB that is different from the first color. In a third image synthesized by overlapping the first image and the second image, the image classification unit sets the defect category of the area displayed in the first color to a bright defect candidate, and sets the defect category of the area displayed in the second color to a dark defect candidate.

[0016] A seventh aspect of the present invention is an inspection support device of any one of aspects 1 to 3 (or any one of aspects 1 to 6), wherein the image classification unit classifies each defect candidate image based on both the defect category and the lethality of the defect candidate contained in each defect candidate image. In the classification display data, two or more defect candidate images corresponding to one defect category can be displayed based on the magnitude relationship of lethality.

[0017] Mode eight of the present invention is an inspection assisting method for assisting the inspection of a substrate having a pattern on its surface, the inspection assisting method comprising the following steps: a) storing a plurality of defect candidate images detected through a single inspection of the substrate image; and b) classifying the plurality of defect candidate images based on rules, classifying the plurality of defect candidate images into a plurality of classified image groups based on the defect category or lethality of the defect candidates contained in each defect candidate image, and generating classified display data capable of displaying the plurality of defect candidate images in a display unit according to each classified image group.

[0018] A ninth aspect of the present invention is a computer-readable recording medium having a program recorded thereon, wherein the program enables a computer to perform inspection assistance for a substrate having a pattern on its surface, and enables the computer to perform the following steps: a) storing a plurality of defect candidate images detected through a single inspection of the substrate image; and b) classifying the plurality of defect candidate images based on rules, classifying the plurality of defect candidate images into a plurality of classified image groups based on the defect category or lethality of the defect candidates contained in each defect candidate image, and generating classified display data capable of displaying the plurality of defect candidate images in a display unit according to each classified image group.

[0019] The above-mentioned object and other objects, features, aspects and advantages will become clear from the following detailed description of the present invention with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 This is a diagram showing an inspection support device according to an embodiment together with a primary inspection device.

[0021] Figure 2 It is a diagram showing the structure of the inspection support device.

[0022] Figure 3 It is a diagram showing the functional structure of the inspection support device.

[0023] Figure 4 This is a diagram showing the flow of generating a learned model in the inspection support device.

[0024] Figure 5 A graph showing defect candidates of multiple defect categories.

[0025] Figure 6 FIG. 1 is a diagram showing a display on which a defect candidate image is displayed.

[0026] Figure 7 FIG. 1 is a diagram showing a display on which a defect candidate image is displayed.

[0027] Figure 8 FIG. 1 is a diagram showing a display on which a defect candidate image is displayed.

[0028] Fig. 9 is a diagram showing a defect candidate image.

[0029] Fig.10 is a diagram showing a defect candidate image.

[0030] Fig.11A is a diagram showing a defect candidate image.

[0031] Fig. 11B is a diagram showing a defect candidate image.

[0032] Fig. 11C is a diagram showing a defect candidate image. DETAILED DESCRIPTION

[0033] Figure 1 1 is a diagram showing an inspection auxiliary device 1 according to an embodiment of the present invention together with a primary inspection device 7. The inspection auxiliary device 1 is, for example, a device that assists in the appearance inspection of a printed wiring substrate 9 (hereinafter, also referred to as "substrate 9") before electronic components are mounted. A pattern (for example, a wiring pattern or an electrode pattern made of copper) is formed on the surface of the substrate 9, which is the inspection object of the appearance inspection. In the inspection auxiliary device 1, for example, a learned model used in defect detection of the substrate 9 based on AI inspection is created.

[0034] The primary inspection device 7 is a device that performs a primary inspection on the substrate 9 to detect defect candidates, and sends a defect candidate image containing the defect candidates to the inspection auxiliary device 1. The primary inspection device 7 includes an image acquisition device 72 that captures the substrate 9 and a computer 73. The computer 73 is a processing device including a computing unit. The computer 73 also controls the overall operation of the primary inspection device 7. The image acquisition device 72 includes a capturing unit 74, a workbench 75 that holds the substrate 9, and a workbench moving mechanism 76.

[0035] The imaging unit 74 emits illumination light toward the substrate 9, receives reflected light from the substrate 9 through an imaging sensor or the like, and thereby captures the substrate 9 to obtain an image. This image is, for example, a grayscale image of the entire area on the substrate 9 that is the inspection object, and is also referred to as a "substrate image" in the following description. In addition, the substrate image obtained by the imaging unit 74 may also be, for example, a multi-grayscale color image. The workbench 75 is a holding unit that holds the substrate 9. The workbench 75 includes, for example, a vacuum chuck that adsorbs and holds the lower surface of the substrate 9. The workbench moving mechanism 76 is composed of a ball screw, a guide rail, a motor, etc., and moves the workbench 75 relative to the imaging unit 74.

[0036] The computer 73 has a general computer system configuration including a CPU, a GPU, a ROM, a RAM, a fixed disk, a display, a keyboard, a mouse, a reading device, a communication unit, etc. In the primary inspection device 7, the computer 73 controls the stage moving mechanism 76 and the imaging unit 74, whereby the stage 75 moves in the horizontal direction to image a desired area on the substrate 9. The substrate image obtained by the imaging unit 74 is sent to the computer 73.

[0037] The computer 73 inspects the substrate image and detects defect candidates. In the following description, the inspection performed by the inspection device 7 is also referred to as "one inspection". In one inspection, for example, an image representing a portion of the substrate 9 (i.e., a portion of the captured image) is taken as the object of one inspection, and the inspection is repeatedly performed on the entire region of the substrate 9 that becomes the inspection object, thereby performing one inspection of the entire captured image. In the following description, an image that is the object of one inspection in one inspection (i.e., an image representing a portion of the substrate 9) is also referred to as an "inspected image".

[0038] In one inspection, for example, a rule-based inspection is performed. Rule-based inspection refers to an inspection method in which certain rules are set for the shape of the pattern on the substrate 9, and the parts of the pattern on the substrate 9 that deviate from the rules are judged as defects. Various methods are known. In this one inspection, for example, a predetermined feature quantity is calculated from each part of the pattern contained in the inspected image through a judgment algorithm pre-made based on the design rules, and the feature quantity is compared with a threshold value for distinguishing qualified products and defect candidates. Then, based on the comparison result of the above-mentioned feature quantity and the threshold value, it is determined whether the inspected image is a defect candidate image containing defect candidates (that is, whether there are defect candidates in the inspected image). The feature quantity is calculated based on, for example, the line width of the linear pattern element contained in the pattern, the end shape of the pattern element, etc. In addition, in one inspection, the inspected image may also be subjected to predetermined image processing before extracting the above-mentioned feature quantity.

[0039] An inspected image determined to contain a defect candidate in a single inspection is sent from the primary inspection device 7 to the inspection auxiliary device 1 and stored in the inspection auxiliary device 1 as a defect candidate image. In the primary inspection device 7, for example, each time a defect candidate image is detected, the defect candidate image is sent to the inspection auxiliary device 1. In the inspection auxiliary device 1, a plurality of defect candidate images are stored as data arranged in the order in which they are detected in the primary inspection device 7 (hereinafter also referred to as "defect candidate group data"). Alternatively, in the primary inspection device 7, a plurality of defect candidate images detected in a single inspection may also be stored in the computer 73 as defect candidate group data arranged in the order in which they are detected. In this case, after a single inspection of all inspected images is completed, the defect candidate group data is sent from the primary inspection device 7 to the inspection auxiliary device 1. The number of defect candidate images contained in the defect candidate group data is, for example, more than 10,000. In addition, an inspected image determined not to contain a defect candidate in a single inspection is stored, for example, in the computer 73 as a qualified product image.

[0040] Figure 2 1 is a diagram showing the structure of the inspection auxiliary device 1. The inspection auxiliary device 1 has a general computer system structure including a CPU 31, a ROM 32, a RAM 33, a fixed disk 34, a display 35, an input unit 36, a reading device 37, a communication unit 38, a GPU 39, and a bus 30. The CPU 31 performs various calculations. The GPU 39 performs various calculations related to image processing. The ROM 32 stores basic programs. The RAM 33 stores various information. The fixed disk 34 stores information. The display 35 is a display unit that displays various information such as images. The input unit 36 ​​has a keyboard 36a and a mouse 36b that receive input from an operator.

[0041] The reading device 37 reads information from a computer-readable recording medium 81 such as an optical disk, a magnetic disk, a magneto-optical disk, or a memory card. The display 35, the keyboard 36a, the mouse 36b, and the reading device 37 are connected to the bus 30 via the interface I / F. The communication unit 38 sends and receives signals to and from other components of the inspection support device 1 and external devices. The bus 30 is a signal circuit that connects the CPU 31, the GPU 39, the ROM 32, the RAM 33, the fixed disk 34, the display 35, the input unit 36, the reading device 37, and the communication unit 38.

[0042] In the inspection support device 1, the program 811 is read from the recording medium 81 via the reading device 37 in advance and stored in the fixed disk 34. The program 811 may also be stored in the fixed disk 34 via a network. The CPU 31 and the GPU 39 perform calculation processing using the RAM 33 and the fixed disk 34 according to the program 811. The CPU 31 and the GPU 39 function as a calculation unit. In addition to the CPU 31 and the GPU 39, other structures that function as a calculation unit may also be used.

[0043] Figure 3 The functional structure of the inspection support device 1 is realized by executing arithmetic processing and the like according to the program 811. These functional structures include a storage unit 301, an image classification unit 302, a temporary setting unit 303, a display control unit 304, an input receiving unit 305, a learning data set generation unit 306, and a learning unit 307. All or part of these functions may be realized by a dedicated circuit. In addition, these functions may be realized by a plurality of computers. Figure 3 The image classification unit 302, temporary setting unit 303, display control unit 304, input receiving unit 305, learning data set generation unit 306 and learning unit 307 in the functional structure shown are implemented by CPU 31, GPU 39, ROM 32, RAM 33, fixed disk 34 and their peripheral structures. In addition, the storage unit 301 is mainly implemented by RAM 33 and fixed disk 34.

[0044] Next, refer to Figure 4 The generation process of the above-mentioned learned model in the inspection support device 1 is described in detail. Figure 3 The storage unit 301 shown stores data from a primary inspection device 7 (see Figure 1 ) is sent to the above-mentioned defect candidate group data (i.e., data of multiple defect candidate images) to the inspection auxiliary device 1 (step S11). In the storage unit 301, in addition to the defect candidate group data, various information related to the inspection of the substrate 9 is also pre-stored. In the defect candidate group data, as described above, the multiple defect candidate images are arranged in the order detected by the primary inspection device 7. Therefore, in the defect candidate group data, the multiple defect candidate images are randomly (i.e., disorderly) arranged with respect to the defect category representing the defect candidate type and the fatality of the defect candidate.

[0045] Next, the image classification unit 302 reads the defect candidate group data stored in the storage unit 301, and classifies each defect candidate image for the above-mentioned multiple defect candidate images included in the defect candidate group data. In the image classification unit 302, each defect candidate image is classified based on the defect type and / or fatality of the defect candidate included in each defect candidate image.

[0046] Figure 5 61 shows examples of defect candidates of multiple defect categories included in the defect candidate image 61. Figure 5 In the figure, parallel oblique lines are marked on a plurality of pattern elements 611 to 613 included in the defect candidate image 61. In addition, in the actual defect candidate image 61, the pattern elements 611 to 613 are bright areas with higher brightness than the surrounding background area 614, and the background area 614 is a dark area with lower brightness than the pattern elements 611 to 613. In addition, various changes can be made to the defect classification method of the defect candidate shown below and the name of the defect classification.

[0047] Defect candidate D11 is an "open defect (i.e., broken wire)" in which the linear pattern element 611 is completely interrupted in the length direction. Defect candidates D12 to D14 are "defects" in which a portion of the linear pattern element 612 is missing in the width direction. Defect candidate D15 is a "dropout" in which a portion of the solid pattern element 613 is missing in the form of dots. Defect candidates D11 to D15 are defect candidates in which a pattern element does not exist at a position where the pattern element should exist. Defect candidates D11 to D15 are defect candidates with lower brightness than pattern elements 611 to 613, and are hereinafter also referred to as "dark defect candidates."

[0048] Defect candidate D21 is a "short circuit defect (i.e., short circuit)" in which pattern element 611 and pattern element 612, which should not be connected, are connected. Defect candidates D22 to D24 are "protrusions" in which unnecessary convex portions in pattern element 611 protrude in the width direction. Defect candidate D25 is a "foreign matter" such as isolated point-shaped unnecessary pattern elements in the background area 614. Defect candidates D21 to D25 are defect candidates in which a certain pattern element exists at a position where there should not be any pattern element. Defect candidates D21 to D25 are defect candidates with higher brightness than the background area 614, and are also referred to as "bright defect candidates" below.

[0049] In addition, Figure 5 In order to facilitate understanding of the shape, features, etc. of each defect candidate, a plurality of defect candidates D11 to D15 and D21 to D25 are depicted in one defect candidate image 61 , but the number of defect candidates included in the actual defect candidate image 61 is, for example, one.

[0050] The lethality of a defect candidate is an indicator indicating the degree of adverse effect of the defect candidate on the performance of the substrate 9, etc. For example, the lethality of defect candidates D11 (defect type: open defect) and defect candidate D21 (defect type: short defect) is high, and the lethality of defect candidates D15 (defect type: detachment) and defect candidate D25 (defect type: foreign matter) is low. The lethality of defect candidates D12 to D14 (defect type: defect) is medium, and in defect candidates D12 to D14, for example, the lethality increases as the defect portion of the pattern element 612 becomes larger (i.e., in the order of defect candidates D14, D13, and D12). The lethality of defect candidates D22 to D24 (defect type: protrusion) is medium, and in defect candidates D22 to D24, for example, the lethality increases as the protrusion protruding from the pattern element becomes larger (i.e., in the order of defect candidates D24, D23, and D22).

[0051] In addition, the above-mentioned classification of the lethality of the defect candidates can be changed in various ways. For example, in the defect candidate D15 (falling off), two or more classifications based on the magnitude relationship of lethality can be set in such a way that the lethality increases as the defect portion increases. In addition, in the defect candidate D25 (foreign matter), two or more classifications based on the magnitude relationship of lethality can be set in such a way that the lethality increases as the foreign matter increases in a similar manner.

[0052] In the image classification unit 302, the above classification of the defect candidate images is implemented by classifying the multiple defect candidate images included in the defect candidate group data based on rules. In the rule-based classification, certain rules are set for the shapes of pattern elements in the defect candidate images, and the defect type and fatality of the defect candidate are determined based on the matching degree with the rules. Various methods are known as specific methods of rule-based classification.

[0053] In the rule-based classification, for example, a predetermined feature quantity is calculated from each part of the pattern element included in the defect candidate image by a judgment algorithm pre-generated based on the design rules, and the feature quantity is compared with a threshold value for distinguishing the predetermined defect candidate. Then, based on the comparison result between the feature quantity and the threshold value, the defect category and lethality of the defect candidate included in the inspected image are determined. The feature quantity is obtained based on, for example, the size of the defect candidate, the distance between the defect candidate and the pattern element adjacent to the defect candidate, etc. In addition, in the rule-based classification, the defect candidate image may also be subjected to predetermined image processing before extracting the feature quantity.

[0054] In this embodiment, the image classification unit 302 classifies the plurality of defect candidate images contained in the defect candidate group data into a plurality of classification image groups set based on both the defect category and the fatality of the defect candidates contained in each defect candidate image by performing the above-mentioned rule-based classification on the plurality of defect candidate images (step S12). Specifically, 10 classification image groups G11 to G15 and G21 to G25 corresponding to the above-mentioned defect candidates D11 to D15 and defect candidates D21 to D25 are set, and each defect candidate image is classified into one of the classification image groups G11 to G15 and G21 to G25.

[0055] In the image classification unit 302, one defect candidate image is read out in sequence according to the arrangement order of the plurality of defect candidate images in the defect candidate group data (i.e., according to the order detected in the primary inspection device 7), and classified into a plurality of classification image groups G11 to G15, G21 to G25 according to both the defect category and the fatality of the defect candidate contained in the one defect candidate image. Then, display data (hereinafter also referred to as "classification display data") is generated that can display the plurality of defect candidate images contained in the defect candidate group data for each classification image group G11 to G15, G21 to G25 on a display unit such as the display 35 (step S13). In addition, in the image classification unit 302, the defect candidate images can also be classified into the classification image groups G11 to G15, G21 to G25 based on either the defect category or the fatality of the defect candidate.

[0056] Next, the temporary setting unit 303 sets a temporary authenticity evaluation for the defect candidate images included in each of the classification image groups G11 to G15, G21 to G25 based on the defect categories and fatalities corresponding to the classification image groups G11 to G15, G21 to G25, respectively (step S14). The temporary authenticity evaluation refers to the result of a temporary determination of whether the defect candidate included in the defect candidate image is a defect with a high probability of becoming a quality problem of the substrate 9 (hereinafter also referred to as a "true defect") or a false alarm with a low probability. In the temporary setting unit 303, the same temporary authenticity evaluation is uniformly set for all defect candidate images included in one classification image group.

[0057] For example, the temporary authenticity evaluation of each defect candidate image included in the classification image groups G11 and G21 is set to "true defect". That is, when the defect category of the defect candidate is "open defect (D11)" or "short defect (D21)" with high lethality, the temporary authenticity evaluation of the defect candidate image including the defect candidate is set to "true defect". In addition, the temporary authenticity evaluation of each defect candidate image included in the classification image groups G15 and G25 is set to "false alarm". That is, when the defect category of the defect candidate is "fall-off (D15)" or "foreign matter (D25)" with low lethality, the temporary authenticity evaluation of the defect candidate image including the defect candidate is set to "false alarm".

[0058] The temporary authenticity evaluation of each defect candidate image included in the classification image groups G12 to G14, G22 to G24 is set to "true defect" or "false report" based on the lethality of the defect candidate of the defect candidate image included in each classification image group. That is, when the defect category of the defect candidate is "defect (D12 to D14)" or "protrusion (D22 to D24)" with a medium lethality, the temporary authenticity evaluation of the defect candidate image included in each classification image group is set to "true defect" or "false report" based on the lethality corresponding to each classification image group. For example, the temporary authenticity evaluation of each defect candidate image included in the classification image groups G12 to G13, G22 to G23 with a relatively high lethality among the classification image groups G12 to G14, G22 to G24 is set to "true defect", and the temporary authenticity evaluation of each defect candidate image included in the classification image groups G14 and G24 with a relatively low lethality is set to "false report".

[0059] The temporary authenticity evaluation of each defect candidate image set in the temporary setting unit 303 is associated with each defect candidate image of the classified display data generated by the image classification unit 302. In addition, in the temporary setting unit 303, the temporary authenticity evaluation of the defect candidate image may be set based on either the defect category or the fatality corresponding to the classified image groups G11 to G15 and G21 to G25.

[0060] Next, the display control unit 304 controls the display 35 and the like based on the classified display data generated by the image classification unit 302, and displays the defect candidate image for each of the classified image groups G11 to G15, G21 to G25 on the display 35 (step S15). Specifically, a plurality of defect candidate images included in one of the classified image groups G11 to G15, G21 to G25 are sequentially displayed on the display 35. Switching from one defect candidate image to the next defect candidate image on the display 35 is performed, for example, by an operator operating the inspection assisting device 1.

[0061] Then, when the display of all defect candidate images included in the one classified image group is completed, the defect candidate images included in the next classified image group are sequentially displayed on the display 35. In the inspection auxiliary device 1, by performing this process on all the classified image groups G11 to G15 and G21 to G25, all defect candidate images included in the defect candidate group data are sequentially displayed. In addition, the number of defect candidate images displayed on the display 35 at one time may be one, or may be two or more. In the present embodiment, it is assumed that the number of defect candidate images 61 displayed on the display 35 at one time is one for explanation.

[0062] like Figure 6 As shown, when a defect candidate image 61 is displayed on the display 35, the defect category of the defect candidate contained in the defect candidate image 61 (i.e., the defect category assigned to the defect candidate image 61 by the image classification unit 302, i.e., any one of "open circuit defect", "defect", "detachment", "short circuit defect", "protrusion" and "foreign matter") is displayed in a cell 62 adjacent to the defect candidate image 61. Furthermore, in a cell 63 adjacent to the defect candidate image 61, a temporary authenticity evaluation (i.e., true defect or false alarm) associated with the defect candidate image 61 by the temporary setting unit 303 is displayed. Figure 6 In the example shown, “open defect” as the defect type of the defect candidate D11 is displayed in cell 62 , and “true defect” which is a true or false evaluation temporarily set for the classified image group G11 including the defect candidate D11 is displayed in cell 63 .

[0063] When defect candidate image 61 is displayed on display 35 , the operator visually observes defect candidate image 61 on display 35 and checks whether the defect type in cell 62 and the provisional authenticity evaluation in cell 63 displayed together with defect candidate image 61 are correct.

[0064] For example, if the operator determines that the defect type displayed in the unit 62 is wrong, the operator changes the display content in the unit 62. Specifically, Figure 7 As shown in the example, the operator moves the mouse cursor on the display 35 and places it on the cell 62, and clicks the mouse button, thereby displaying the pull-down menu 64. The operator changes the defect category assigned to the defect candidate image 61 by the image classification unit 302 by selecting a defect category from the pull-down menu 64. On the other hand, when the operator determines that the defect category in the cell 62 is correct, the cell 62 is not particularly operated. In addition, the change of the defect category by the operator is not limited to the above example, and can be performed by various methods.

[0065] In addition, for example, when the operator determines that the temporary authenticity evaluation displayed in the unit 63 is wrong, the operator changes the display content in the unit 63. Specifically, Figure 8 As shown in the example, the operator moves the mouse cursor on the display 35 to place it on the cell 63 and clicks the mouse button, thereby displaying the pull-down menu 65. The operator changes the temporary authenticity evaluation given to the defect candidate image 61 by the temporary setting unit 303 by selecting one of true defect and false report from the pull-down menu 65. On the other hand, when the operator determines that the temporary authenticity evaluation in the cell 63 is correct, the cell 63 is not particularly operated. In addition, the change of the authenticity evaluation by the operator is not limited to the above example, and can be performed by various methods.

[0066] When the defect category and authenticity evaluation of the defect candidate image 61 displayed on the display 35 are examined and the defect category and authenticity evaluation are changed as needed, the operator places the mouse cursor on Figure 6 The operator places the mouse button on the decision button 66 displayed on the lower side of the cells 62 and 63. Thus, the defect category displayed on the cell 62 and the authenticity evaluation displayed on the cell 63 are input, and the input receiving unit 305 accepts the input (step S16). That is, the operation of the operator clicking the decision button 66 is an input operation of inputting the defect category of the defect candidate image 61 and the evaluation of the true defect or false report for the defect candidate image 61 via the input unit 36. In addition, the method of the input operation can be changed in various ways, such as input via the keyboard 36a. In addition, when the operator clicks the decision button 66, the defect candidate image 61 displayed on the display 35 is switched, and the next defect candidate image in the classified display data is displayed.

[0067] In the inspection assisting device 1, for a plurality of defect candidate images contained in the classified display data (i.e., each defect candidate image classified into the classified image groups G11 to G15, G21 to G25), the above-mentioned defect category and authenticity evaluation are studied in sequence by the operator, and the defect category and authenticity evaluation are changed as needed, and the results are accepted by the input accepting unit 305.

[0068] As described above, in the inspection assisting device 1, a plurality of defect candidate images are displayed on the display 35 for each of the classified image groups G11 to G15 and G21 to G25. Specifically, for example, a plurality of defect candidate images included in the classified image group G11 are displayed in sequence, and then a plurality of defect candidate images included in the classified image group G12 are displayed in sequence. Next, a plurality of defect candidate images included in the classified image group G13 are displayed in sequence, a plurality of defect candidate images included in the classified image group G14 are displayed in sequence, and a plurality of defect candidate images included in the classified image group G15 are displayed in sequence. Thereafter, a plurality of defect candidate images included in the classified image group G21 are displayed in sequence, and then a plurality of defect candidate images included in the classified image group G22 are displayed in sequence. Next, a plurality of defect candidate images included in the classified image group G23 are displayed in sequence, a plurality of defect candidate images included in the classified image group G24 are displayed in sequence, and a plurality of defect candidate images included in the classified image group G25 are displayed in sequence.

[0069] That is, in the inspection auxiliary device 1, a plurality of defect candidate images are displayed on the display 35 in a state classified based on both defect categories and lethality. Therefore, compared with a case where a plurality of defect candidate images are randomly displayed with respect to defect categories and lethality (for example, a case where a plurality of defect candidate images are displayed in the order detected by the primary inspection device 7), the operator can intensively and continuously observe a plurality of defect candidate images with the same defect categories and lethality, and determine their defect categories and authenticity evaluations. Therefore, it is possible to suppress changes in the operator's determination criteria with the passage of time, etc., and to suppress the occurrence of deviations in the determination of the defect categories and authenticity evaluations of the defect candidate images.

[0070] When step S16 for all defect candidate images included in the classified display data is completed, the defect category and authenticity evaluation are matched with each defect candidate image of the classified display data based on the information received by the input receiving unit 305 by the learning data set generating unit 306. That is, for each defect candidate image, when the input receiving unit 305 receives an input of a defect category different from the defect category determined in the image classifying unit 302, the defect category received by the input receiving unit 305 is matched with the defect candidate image in the learning data set generating unit 306. On the other hand, in other cases (that is, when the defect category determined in the image classifying unit 302 is not changed), the defect category determined in the image classifying unit 302 is directly matched with the defect candidate image.

[0071] In addition, for each defect candidate image, when the input receiving unit 305 receives an input of an authenticity evaluation different from the temporary authenticity evaluation set in the temporary setting unit 303, the authenticity evaluation received by the input receiving unit 305 is matched with the defect candidate image in the learning data set generating unit 306. On the other hand, in other cases (i.e., when the temporary authenticity evaluation set in the temporary setting unit 303 has not been changed), the temporary authenticity evaluation set in the temporary setting unit 303 is directly matched with the defect candidate image. Thus, a learning data set for making a learned model used in defect detection of the substrate 9 based on AI inspection is generated (step S17). In addition, the learning data set may also include images of qualified products stored in the computer 73 by the above-mentioned primary inspection device 7.

[0072] When step S17 is completed, the learning unit 307 uses the above-mentioned learning data set to perform machine learning on the initial model to produce the above-mentioned learned model (step S18). The machine learning is performed, for example, by deep learning using a neural network. The learning based on deep learning is performed, for example, using ResNet. In addition, the machine learning can also be performed by methods other than deep learning.

[0073] The learned model generated by the learning unit 307 is sent to a secondary inspection device (not shown) that performs AI inspection on defect candidate images to detect real defects, and is used for AI inspection in the secondary inspection device. In addition, the function of the secondary inspection device can be realized by a computer (refer to Figure 2 ) to achieve it.

[0074] Next, regarding the classification of defect candidate images in step S12, a specific example of a classification method based on the fatality of defect candidates is described. Figure 5 When classifying the defect candidates D12 to D14 whose defect category is "defective", the image classification unit 302 obtains the feature quantity of each of the defect candidates D12 to D14, and determines the fatality of the defect candidates D12 to D14 based on the feature quantity. Then, by comparing the fatality with a predetermined threshold, the above-mentioned rule-based classification of the defect candidate images containing the defect candidates D12 to D14 is performed.

[0075] For example, Fig. 9The above-mentioned feature quantity of the defect candidate D12 shown includes the size of the defect candidate D12 (i.e., the area of ​​the portion of the pattern element 612 that is missing as the defect candidate D12). In addition, the feature quantity includes the width B11 of the pattern element 612 at the position of the defect candidate D12 (i.e., the full width of the pattern element 612 in the width direction when the position where the defect candidate D12 exists in the pattern element 612 is assumed to be the position where the defect candidate D12 exists).

[0076] The lethality determined based on the feature quantity increases as the size of the defect candidate D12 increases. In addition, the lethality determined based on the feature quantity increases as the ratio of the width B12 (i.e., the maximum width in the width direction) of the defect candidate D12 to the width B11 of the pattern element 612 at the position of the defect candidate D12 increases. In addition, the width B11 and the width B12 are not necessarily limited to the width in the width direction (i.e., the direction perpendicular to the length direction) of the pattern element 612, and can also be the width in any direction intersecting the length direction.

[0077] Furthermore, in addition to or instead of the size of the defect candidate D12 and the width B11 of the pattern element 612 at the position of the defect candidate D12, the lethality of the defect candidate D12 may be determined based on the density of the defect candidate D12. In this case, as the density of the defect candidate D12, which is a dark defect candidate, is closer to the density of the background region 614 than the density of the pattern element 612, the lethality of the defect candidate D12 becomes higher. On the other hand, as the density of the defect candidate D12 is closer to the density of the pattern element 612 than the density of the background region 614, the lethality of the defect candidate D12 becomes lower.

[0078] The feature values ​​of defect candidates D13 and D14 and the lethality determined based on the feature values ​​are the same as those of defect candidate D12. Therefore, for defect candidates D12 to D14 whose defect category is "defect", the lethality can be determined with high accuracy by considering the impact of the defect candidates D12 to D14 on the performance of the substrate 9.

[0079] In addition, based on the fatality, defect candidates D22 to D24 of the defect type "protrusion" (see Figure 5 ) is classified, the image classification unit 302 obtains the feature value of each defect candidate D22 to D24, and determines the fatality of the defect candidates D22 to D24 based on the feature value. Then, by comparing the fatality with a predetermined threshold, the above-mentioned rule-based classification of the defect candidate images containing the defect candidates D22 to D24 is performed.

[0080] For example, Fig.10The above-mentioned feature quantity of the defect candidate D22 shown includes the size of the defect candidate D22 (i.e., the area of ​​the defect candidate D22 protruding from the pattern element 611). In addition, the feature quantity includes the width B21 of the background area 614 at the position of the defect candidate D22 (i.e., the width in the width direction of the background area 614 adjacent to the pattern element 611 on the side where the defect candidate D22 protrudes at the position where the defect candidate D22 exists on the pattern element 611, which is the full width in the width direction when it is assumed that the defect candidate D22 does not exist).

[0081] The lethality determined based on the feature quantity becomes higher as the size of the defect candidate D22 becomes larger. In addition, the lethality determined based on the feature quantity increases as the width B22 (i.e., the maximum width in the width direction) of the defect candidate D22 relative to the width B21 (i.e., the maximum width in the width direction) of the background area 614 at the position of the defect candidate D22. Fig. 9 In the example shown, the ratio of the spacing between pattern element 611 and pattern element 612 becomes larger and higher. In addition, width B21 and width B22 are not necessarily limited to the width direction of pattern element 611 (i.e., the direction perpendicular to the length direction), and can also be the width of any direction intersecting the length direction.

[0082] Furthermore, in addition to or instead of the size of the defect candidate D22 and the width B21 of the background region 614 at the position of the defect candidate D22, the lethality of the defect candidate D22 may be determined based on the density of the defect candidate D22. In this case, as the density of the defect candidate D22, which is a bright defect candidate, is closer to the density of the pattern element 611 than the density of the background region 614, the lethality of the defect candidate D22 becomes higher. On the other hand, as the density of the defect candidate D22 is closer to the density of the background region 614 than the density of the pattern element 611, the lethality of the defect candidate D22 becomes lower.

[0083] The feature values ​​of defect candidates D23 and D24 and the lethality determined based on the feature values ​​are the same as those of defect candidate D22. Therefore, for defect candidates D22 to D24 whose defect category is "protrusion", the lethality can be determined with high accuracy by considering the impact of defect candidates D22 to D24 on the performance of substrate 9.

[0084] In addition, the lethality of the defect candidate does not necessarily have to be determined based on the size, concentration, etc. of the defect candidate as exemplified above, but may be determined based on other feature quantities of the defect candidate.

[0085] In the above example, in the image classification unit 302, defect candidates are classified into six defect categories, namely, "open defect", "defect", "detachment", "short defect", "protrusion" and "foreign matter", but not limited to this. For example, defect candidates may also be classified into the above-mentioned "bright defect candidate" and "dark defect candidate". Alternatively, after classifying the defect candidates into either "bright defect candidate" or "dark defect candidate", the defect candidate classified as "bright defect candidate" may be classified into either "open defect", "defect" or "detachment", and the defect candidate classified as "dark defect candidate" may be classified into either "short defect", "protrusion" or "foreign matter".

[0086] When classifying defect candidates as “bright defect candidates” or “dark defect candidates”, for example, Fig.11A As shown, Figure 5 The defect candidate image 61 shown is converted into a first image 67 expressed by a first color (in this embodiment, "green (G)") which is any one of RGB colors. Fig.11A In FIG. 6 , the areas colored green in the first image 67 (ie, pattern elements 611 to 613 and defect candidates D21 to D25) are marked with parallel oblique lines. Fig.11A , the background area 614 and defect candidates D11 to D15 are not colored.

[0087] In addition, if Fig. 11B As shown, it will be Figure 5 The reference image corresponding to the defect candidate image 61 shown is transformed into a second color in RGB that is different from the first color (in the present embodiment, "red (R)"), and a second image 68 represented by the second color is generated. The reference image is an image that does not have the defect candidates included in the defect candidate image 61. In other words, the reference image is an image in which all the defect candidates D11 to D15 and D21 to D25 included in the defect candidate image 61 are removed (that is, an inspected image that does not have any defect candidates at all). The reference image is pre-stored in the storage unit 301, for example. Fig. 11B In FIG. 6A , the areas colored red in the second image 68 (i.e., pattern elements 611 to 613) are marked with parallel oblique lines. Fig. 11B , the background area 614 is not colored.

[0088] When the first image 67 and the second image 68 are prepared, the first image 67 and the second image 68 are overlapped and synthesized to generate Fig. 11CThe third image 69 is shown. In the third image 69, the areas colored in both the first image 67 and the second image 68 (i.e., pattern elements 611 to 613) are colored in yellow. On the other hand, the areas colored in the first image 67 and not colored in the second image 68 (i.e., defect candidates D21 to D25) are displayed in green, which is the first color, in the third image 69. In addition, the areas not colored in the first image 67 but colored in the second image 68 (i.e., defect candidates D11 to D15) are displayed in red, which is the second color, in the third image 69.

[0089] In the image classification unit 302, the defect category of the area displayed in the first color in the third image 69 (i.e., defect candidates D21 to D25) is set to "bright defect candidate", and the defect category of the area displayed in the second color in the third image 69 (i.e., defect candidates D11 to D15) is set to "dark defect candidate". By this method, the defect candidates in the defect candidate image 61 are easily classified as "bright defect candidate" or "dark defect candidate". In addition, the defect candidates may also be classified as "bright defect candidate" or "dark defect candidate" by methods other than the above.

[0090] In the above example, in step S15, a plurality of defect candidate images are displayed on the display 35 for each of the classified image groups G11 to G15, G21 to G25 (i.e., in a state where they are classified based on both the defect category and the fatality), but the present invention is not limited thereto. For example, in step S15, a plurality of defect candidate images may be displayed on the display 35 in a state where they are classified based on only one of the defect category and the fatality.

[0091] Specifically, when a plurality of defect candidate images are classified based only on the defect category and displayed on the display 35 based on the classification result, for example, defect candidate images including defect candidates D12 to D14 whose defect category is "defect" are displayed in an arbitrary order as defect candidate images classified into the same defect category on the display 35. In addition, defect candidate images including defect candidates D22 to D24 whose defect category is "protrusion" are also displayed in an arbitrary order on the display 35 as defect candidate images classified into the same defect category.

[0092] Alternatively, when a plurality of defect candidate images are classified based only on the lethality and displayed on the display 35 based on the classification result, for example, defect candidate images including defect candidates D11 and D21 with high lethality are displayed in an arbitrary order as defect candidate images classified as having the same level of lethality on the display 35. Also, defect candidate images including defect candidates D15 and D25 with low lethality are displayed in an arbitrary order on the display 35 as defect candidate images classified as having the same level of lethality on the display 35.

[0093] As described above, the inspection assisting device 1 is a device that assists in the inspection of the substrate 9 having a pattern on the surface. The inspection assisting device 1 includes a storage unit 301 and an image classification unit 302. The storage unit 301 stores a plurality of defect candidate images detected by a single inspection of the substrate image. The image classification unit 302 classifies the plurality of defect candidate images based on rules, and classifies the plurality of defect candidate images into a plurality of classified image groups (in the above example, classified image groups G11 to G15, G21 to G25) based on the defect type or fatality of the defect candidates (in the above example, defect candidates D11 to D15, D21 to D25) contained in each defect candidate image. The image classification unit 302 generates classified display data that can display the plurality of defect candidate images in the display unit (in the above example, the display 35) for each classified image group.

[0094] Thus, the operator can continuously observe multiple defect candidate images with the same defect category or fatality in the above display unit. Therefore, when the operator makes a predetermined judgment on the multiple defect candidate images (in the above example, the judgment of the defect category of the defect candidate and the judgment of true defect or false alarm), the change of the operator's judgment standard with the passage of time can be suppressed, and the deviation of the judgment can be suppressed. In other words, the inspection auxiliary device 1 can improve the accuracy of the operator's visual judgment of the defect candidate images.

[0095] As described above, the inspection assisting device 1 preferably further includes a display unit (i.e., display 35), an input receiving unit 305, a learning data set generating unit 306, and a learning unit 307. The display 35 displays a plurality of defect candidate images for each of the above-mentioned classified image groups. The input receiving unit 305 receives input of an evaluation of defects (i.e., true defects) or false reports made by an operator of each defect candidate image displayed in the visual display 35 as an authenticity evaluation of the above-mentioned defect candidate image. The learning data set generating unit 306 generates a learning data set by associating the authenticity evaluation with each defect candidate image in the above-mentioned classified display data. The learning unit 307 uses the learning data set to perform machine learning on the initial model, thereby producing a learned model used in the AI ​​inspection of the substrate image.

[0096] This makes it possible to generate a learning data set with little deviation in the authenticity evaluation teaching by the operator (i.e., high authenticity judgment accuracy), and to create a learned model capable of high-precision authenticity judgment. As a result, the inspection accuracy of AI inspection of substrate images can be improved.

[0097] As described above, the inspection auxiliary device 1 preferably further includes a temporary setting unit 303. The temporary setting unit 303 sets a temporary authenticity evaluation for each defect candidate image included in each classified image group based on the defect category or fatality corresponding to each classified image group. When the input receiving unit 305 receives an input different from the temporary authenticity evaluation, the learning data set generating unit 306 matches the authenticity evaluation received by the input receiving unit 305 with the defect candidate image, and in other cases, matches the temporary authenticity evaluation with the defect candidate image.

[0098] In this way, by temporarily setting the authenticity evaluation by the temporary setting unit 303 before the operator who visually views each defect candidate image inputs the authenticity evaluation, the possibility that the operator inputs the authenticity evaluation only for a part of the defect candidates or does not input the authenticity evaluation at all increases. As a result, the time required for the operator to input the authenticity evaluation can be shortened.

[0099] As described above, in the image classification unit 302, each defect candidate image is preferably classified based on the lethality of the defect candidate contained in each defect candidate image, and the lethality of the defect candidate is determined based on the size of the defect candidate contained in each defect candidate image and the width of the pattern element or background area at the position of the defect candidate. In this way, the lethality can be determined with high accuracy by appropriately considering the influence of the defect candidate on the performance of the substrate 9.

[0100] As described above, it is also preferred that each defect candidate image is classified based on the lethality of the defect candidate contained in the defect candidate image, and the lethality of the defect candidate is determined based on the concentration of the defect candidate contained in each defect candidate image in the image classification unit 302. Therefore, the lethality can be determined with high accuracy by appropriately considering the influence of the defect candidate on the performance of the substrate 9.

[0101] As described above, in the image classification unit 302, it is also preferred to classify each defect candidate image based on the defect category of the defect candidate included in each defect candidate image. Preferably, the image classification unit 302 transforms each defect candidate image into a first image 67 represented by a first color (green (G) in the above example) which is any one of RGB. In addition, the image classification unit 302 generates a second image 68 that represents a reference image that does not have the defect candidate included in each defect candidate image by a second color (red (R) in the above example) different from the first color in RGB. Then, the image classification unit 302 uses the defect category of the area displayed in the first color as a bright defect candidate and the defect category of the area displayed in the second color as a dark defect candidate in the third image 69 synthesized by superimposing the first image 67 and the second image 68. Thereby, the defect candidates in the defect candidate image can be easily classified as "bright defect candidates" or "dark defect candidates".

[0102] As described above, it is also preferred that each defect candidate image is classified based on the defect category and fatality of the defect candidate contained in each defect candidate image in the image classification unit 302, and in the classification display data, two or more defect candidate images corresponding to one defect category can be displayed based on the magnitude relationship of fatality. In this way, by collectively displaying multiple defect candidate images classified into the same defect category according to each fatality (for example, in order from large to small fatality), it is possible to further suppress the occurrence of deviations in the operator's judgment. That is, the operator's visual judgment accuracy of the defect candidate images can be further improved.

[0103] The above-mentioned inspection assisting method includes the following steps: storing a plurality of defect candidate images detected by a single inspection of a substrate image (step S11); and classifying the plurality of defect candidate images based on rules, classifying the plurality of defect candidate images into a plurality of classified image groups (in the above-mentioned example, classified image groups G11 to G15, G21 to G25) based on the defect category or fatality of the defect candidates (in the above-mentioned example, defect candidates D11 to D15, D21 to D25) contained in each defect candidate image, and generating classified display data (steps S12 to S13) that can display the plurality of defect candidate images in each classified image group on a display unit (in the above-mentioned example, display 35). Thus, as described above, the operator's visual judgment accuracy of the defect candidate images can be improved.

[0104] The computer-readable program 811 causes the computer to execute the following steps: a step of storing a plurality of defect candidate images detected by a single inspection of the substrate image (step S11); and a step of classifying the plurality of defect candidate images based on rules, classifying the plurality of defect candidate images into a plurality of classified image groups (in the above example, classified image groups G11 to G15, G21 to G25) based on the defect type or fatality of the defect candidates (in the above example, defect candidates D11 to D15, D21 to D25) contained in each defect candidate image, and generating classified display data that can display the plurality of defect candidate images in each classified image group on a display unit (in the above example, display 35) (steps S12 to S13). Thus, as described above, the operator's visual judgment accuracy of the defect candidate images can be improved.

[0105] Various changes can be made to the above-mentioned inspection support device 1 , inspection support method, and program 811 .

[0106] For example, it is not necessary to set the temporary authenticity evaluation for each defect candidate image in step S14. In this case, in step S16, the operator performs the authenticity evaluation for all defect candidate images and inputs the evaluation results. In this case, the temporary setting unit 303 can be omitted.

[0107] In the above-mentioned step S16, it is not necessary for the operator to be able to change the defect type.

[0108] The defect candidate images of the classified image groups that are classified as part of the above-mentioned multiple classified image groups G11 to G15, G21 to G25 may not be included in the classified display data. For example, each defect candidate image included in the classified image groups G11 and G21 (that is, a defect candidate image including an open defect or a short defect) may not be included in the classified display data. In this case, each defect candidate image included in the classified image groups G11 and G21 will not become a setting object for the temporary authenticity evaluation of the temporary setting unit 303, and there is no need to display it on the display 35 for the operator to perform authenticity evaluation. Each defect candidate image included in the classified image groups G11 and G21 is sent directly from the image classification unit 302 to the learning data set generation unit 306, and is included in the learning data set as a defect candidate image containing a true defect with a high degree of lethality.

[0109] The above-mentioned learned model does not necessarily need to be generated in the inspection support device 1. For example, the inspection support device 1 may send the classified display data generated by the image classification unit 302 to a verification device that is a device different from the inspection support device 1. In the verification device, a plurality of defect candidate images included in the classified display data are sequentially displayed on a display unit such as a monitor, and an operator visually observes the defect candidate images on the display unit to make an evaluation of true defects or false reports.

[0110] In this case, the operator can observe multiple defect candidate images with the same defect category or fatality in a concentrated manner in the display unit of the verification device. As a result, the operator's visual judgment accuracy of the defect candidate images can be improved in a substantially similar manner to the above. In addition, when the classified display data is sent to the verification device, the inspection auxiliary device 1 only needs to have the above-mentioned storage unit 301 and image classification unit 302, and other structures (for example, the display unit, the input receiving unit 305, the learning data set generation unit 306, and the learning unit 307, etc.) can be omitted.

[0111] In the primary inspection performed by the primary inspection device 7, in addition to the above-mentioned rule-based inspection, or as a substitute for the above-mentioned rule-based inspection, inspection may also be performed by other methods (for example, AI inspection).

[0112] The substrate 9 is not necessarily limited to a printed wiring board, and may be, for example, a semiconductor substrate, a semiconductor package substrate, a glass substrate for a flat panel display such as a liquid crystal display or a plasma display, a glass substrate for a photomask, or a substrate for a solar cell panel.

[0113] The configurations in the above-described embodiment and various modifications may be appropriately combined unless they are in conflict with each other.

[0114] Although the invention has been described and illustrated in detail, the above description is illustrative and not restrictive. Therefore, it can be said that various modifications or aspects can be adopted without departing from the scope of the present invention.

[0115] Description of Reference Numerals

[0116] 1 Check auxiliary devices

[0117] 9 substrates

[0118] 35 Display

[0119] 61 defect candidate images

[0120] 67 First Image

[0121] 68 Second Image

[0122] 69 Third Image

[0123] 301 Storage Department

[0124] 302 Image Classification Department

[0125] 303 Temporary Setting Department

[0126] 305 Input receiving unit

[0127] 306 Learning Dataset Generation Department

[0128] 307 Learning Department

[0129] 611~613 pattern elements

[0130] 614 Background area

[0131] 811 Program

[0132] D11~D15、D21~D25 defect candidates

[0133] G11~G15, G21~G25 classification image group

[0134] Steps S11 to S18.

Claims

1. An inspection assisting device for assisting inspection of a substrate having a pattern on its surface, characterized in that: The inspection auxiliary device comprises: a storage unit that stores a plurality of defect candidate images detected by one inspection of the substrate image; and An image classification unit classifies the multiple defect candidate images based on rules, classifies the multiple defect candidate images into multiple classification image groups based on the defect category or fatality of the defect candidates contained in each defect candidate image, and generates classification display data that can display the multiple defect candidate images in each classification image group on a display unit.

2. The inspection auxiliary device according to claim 1, characterized in that: The inspection auxiliary device also includes: a display unit configured to display the plurality of defect candidate images for each of the classified image groups; an input receiving unit that receives an input of an evaluation of a defect or false report made by an operator who visually views each defect candidate image displayed on the display unit as an evaluation of the authenticity of each defect candidate image; a learning data set generating unit that generates a learning data set by associating the authenticity evaluation with each of the defect candidate images in the classification display data; and A learning unit performs machine learning on an initial model using the learning data set, thereby creating a learned model for AI inspection of substrate images.

3. The inspection auxiliary device according to claim 2, characterized in that: The inspection auxiliary device further includes a temporary setting unit, which sets a temporary authenticity evaluation for each defect candidate image included in each classified image group based on the defect category or fatality corresponding to each classified image group. When the input receiving unit receives an input different from the temporary authenticity evaluation, the learning data set generating unit matches the authenticity evaluation received by the input receiving unit with the defect candidate image; in other cases, the learning data set generating unit matches the temporary authenticity evaluation with the defect candidate image.

4. The inspection auxiliary device according to any one of claims 1 to 3, characterized in that: In the image classification unit, each defect candidate image is classified based on the fatality of the defect candidate included in each defect candidate image. The criticality of the defect candidate is determined based on the size of the defect candidate included in each defect candidate image and the width of the pattern element or the background area at the position of the defect candidate.

5. The inspection auxiliary device according to any one of claims 1 to 3, characterized in that: In the image classification unit, each defect candidate image is classified based on the fatality of the defect candidate included in each defect candidate image. The lethality of the defect candidate is determined based on the density of the defect candidate included in each defect candidate image.

6. The inspection auxiliary device according to any one of claims 1 to 3, characterized in that: In the image classification unit, each defect candidate image is classified based on the defect category of the defect candidate included in each defect candidate image, The image classification unit transforms each defect candidate image into a first image represented by a first color which is any one of RGB colors, and generates a second image which represents a reference image which does not have the defect candidates included in each defect candidate image by a second color in RGB which is different from the first color. In a third image synthesized by overlapping the first image and the second image, the image classification unit sets the defect category of the area displayed by the first color as a bright defect candidate, and sets the defect category of the area displayed by the second color as a dark defect candidate.

7. The inspection auxiliary device according to any one of claims 1 to 3, characterized in that: In the image classification unit, each defect candidate image is classified based on both the defect type and the fatality of the defect candidate included in each defect candidate image. In the classification display data, two or more defect candidate images corresponding to one defect category can be displayed based on the magnitude relationship of fatality.

8. An inspection assisting method for assisting the inspection of a substrate having a pattern on its surface, characterized in that: The inspection auxiliary method comprises the following steps: storing a plurality of defect candidate images detected through one inspection of the substrate image; and By performing rule-based classification on the multiple defect candidate images, the multiple defect candidate images are classified into multiple classified image groups based on the defect category or fatality of the defect candidates contained in each defect candidate image, and classified display data capable of displaying the multiple defect candidate images according to each classified image group on a display unit is generated.

9. A computer-readable recording medium having a program recorded thereon, the program causing a computer to perform inspection assistance of a substrate having a pattern on a surface thereof, It is characterized in that The program causes the computer to execute the following steps: storing a plurality of defect candidate images detected through one inspection of the substrate image; and By performing rule-based classification on the multiple defect candidate images, the multiple defect candidate images are classified into multiple classified image groups based on the defect category or fatality of the defect candidates contained in each defect candidate image, and classified display data capable of displaying the multiple defect candidate images according to each classified image group on a display unit is generated.

Images