Inspection equipment

Abstract

This inspection device provides a process that improves the reliability of a judgment method using learned results by filtering the image data to be learned based on the operator's judgment during the learning process and approving the judgment result of the learned data. [Solution] The control unit 30 of the inspection device 10 includes a primary determination unit (inspection unit 34) that determines whether the object to be inspected 12 is good or bad using image data, a re-determination unit (inspection unit 34) that determines whether the object to be inspected 12 is good or bad in a different way than the primary determination unit using image data that has been determined to be bad by the primary determination unit, a pre-processing unit (inspection information processing unit 33) that groups and stores image data that has been determined to be bad by the primary determination unit and good by the re-determination unit, and a learning unit (inspection information processing unit 33) that learns the state of the object to be inspected 12 when it is good using image data from the group of image data that satisfies predetermined conditions from among the image data grouped by the pre-processing unit.

Description

Technical Field

[0001] The present invention relates to an inspection apparatus.

Background Art

[0002] In an inspection apparatus for inspecting an electronic circuit board (hereinafter also simply referred to as "board"), inspection is performed on the image data obtained by imaging this board using preset inspection information (inspection rules) (this inspection is called "primary determination processing").

[0003] In the primary determination processing, if the inspection conditions are not set appropriately, the over-detection in which a board that should originally be judged as a good product is judged as a defective product increases. Also, even when the inspection conditions are set appropriately, there are cases where it is judged as a defective product due to a change in the manufacturer of the parts, etc. The board judged as a defective product by the inspection apparatus is again judged as a good or defective product by an operator (a visual judge), but the work of the operator increases due to the increase in over-detection (the inspection performed again visually by the operator according to the result of the primary determination processing is called "visual determination processing"). As a result, the attention of the operator decreases, and a defective product is judged as a good product, which causes defective boards to flow out.

[0004] As an improvement method for the above problems in the primary determination processing, a method of performing inspection using the learning result by AI or the like in the primary determination processing has been proposed (for example, see Patent Document 1). In the method of learning using AI, an inference model is created using the image data of the board acquired by the inspection apparatus, and the board is inspected using this inference model.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] However, when using a judgment method that utilizes the learning results of AI or the like for the initial judgment process, there was a problem in that the accuracy of the judgment by the AI ​​or the like decreased because good product images that should not be used as training subjects were included in the learning process.

[0007] The present invention has been made in view of the above problems, and is an inspection device configured to perform a visual inspection process by an operator on an object (substrate) to be inspected that has been determined to be defective as a result of a primary judgment process, learn the operator's judgment result, and use the learned result to perform an inspection on the object to be inspected that has been determined to be defective in the primary judgment process (this inspection is called the "secondary judgment process"). The objective is to provide an inspection device that improves the reliability of the secondary judgment process, which is a judgment method using the learned result, by providing a process to approve the judgment result of the object to be learned by filtering the image data to be learned based on the results of the visual inspection process by an operator during the learning process. [Means for solving the problem]

[0008] To solve the above problems, the inspection apparatus according to the present invention comprises an imaging unit that images an object to be inspected, and a control unit that inspects the object to be inspected from the image data of the object to be inspected captured by the imaging unit, wherein the control unit comprises a primary determination unit that determines whether the object to be inspected is good or bad using the image data, a re-determination unit that determines whether the object to be inspected is good or bad in a different way than the primary determination unit using the image data that the primary determination unit determined to be bad, a pre-processing unit that groups and stores the image data that the primary determination unit determined to be bad and the re-determination unit determined to be good, and a learning unit that learns the state of the object to be inspected when it is good using the image data of the group that satisfies predetermined conditions from the image data grouped by the pre-processing unit.

[0009] In such an inspection apparatus according to the present invention, it is preferable that the re-determination unit is configured to present the image data to an operator and receive the result of the operator's visual determination.

[0010] Furthermore, in the inspection apparatus according to the present invention, it is preferable that the re-determination unit is configured to make a secondary determination of whether the object to be inspected is good or bad based on the results learned by the learning unit.

[0011] Furthermore, in the inspection apparatus according to the present invention, when the learning unit learns the image data of the group, it is preferable that it generates and stores average image data, in which the average value of each pixel at the same position in the image data within the group is set as the value of the pixel, and dispersed image data, in which the variance value of each pixel at the same position in the image data within the group is set as the value of the pixel, and the re-determination unit calculates the difference between each pixel between the image data learned by the learning unit and the image data to be inspected using the average image data and the dispersed image data.

[0012] Furthermore, in the inspection apparatus according to the present invention, it is preferable that the re-evaluation unit displays the abnormal location on the image data and issues a warning when the result of the secondary evaluation is unsatisfactory.

[0013] Furthermore, in the inspection apparatus according to the present invention, the re-evaluation unit is configured to present the image data to the operator and accept the result of the operator's visual evaluation when the result of the secondary evaluation is poor, and the pre-processing unit preferably groups and stores the image data when it is determined to be good by the visual evaluation.

[0014] Furthermore, in the inspection apparatus according to the present invention, the control unit is configured to switch between a main operation mode and a temporary operation mode as the operation mode, the re-determination unit is configured to present the image data to the operator and accept the result of the operator's visual inspection when the result of the secondary determination is good and the operation mode is the temporary operation mode, and the pre-processing unit preferably groups and stores the image data when it is determined to be good by the visual inspection.

[0015] Furthermore, in the inspection apparatus according to the present invention, it is preferable that the re-evaluation unit stores the image data as missed image data when the result of the visual evaluation is poor and the result of the secondary evaluation is good.

[0016] Furthermore, in the inspection apparatus according to the present invention, it is preferable that the preprocessing unit calculates the similarity between the new image data and already stored image data when storing new image data, and groups image data with a similarity of a predetermined value or higher.

[0017] Furthermore, in the inspection apparatus according to the present invention, it is preferable that the preprocessing unit calculates the similarity using template matching or the phase-limit correlation method.

[0018] Furthermore, in the inspection apparatus according to the present invention, it is preferable that the learning unit learns using the image data of a group when the number of image data belonging to that group exceeds a predetermined number, as a predetermined condition.

[0019] Furthermore, in the inspection apparatus according to the present invention, it is preferable that the learning unit learns the image data of the group using representative image data within the group.

[0020] Furthermore, in the inspection apparatus according to the present invention, it is preferable that the learning unit stores the image data of the group as learned image data when it has learned the image data of the group.

[0021] Also, in the inspection apparatus according to the present invention, when the learning unit learns using the image data of the group, it is preferable to receive a determination by an operator as to whether or not to learn the image data of the group, and to execute learning when a determination to learn is received.

[0022] Also, in the inspection apparatus according to the present invention, when the learning unit receives the determination of the operator, it calculates the similarity of the verification image data with respect to the image data for which it is determined whether or not to learn, and displays, on the screen when the reception is performed, the image data for which it is determined whether or not to learn, the verification image data, and the similarity.

[0023] Also, in the inspection apparatus according to the present invention, when the learning unit displays, on the screen when the reception is performed, the image data for which it is determined whether or not to learn, the verification image data, and the similarity, it preferably displays at least one of a histogram or a graph of the similarity, and when an element of the histogram or the graph is selected, it displays the verification image data corresponding to the element.

Advantages of the Invention

[0024] According to the inspection apparatus of the present invention, in the learning process, by filtering the determination result of the operator on the image data to be learned, a process for approving the determination result of the learning target is provided, and thereby, the reliability of the determination method using the learning result can be improved.

Brief Description of the Drawings

[0025] [Figure 1] It is an explanatory diagram for explaining the configuration of the inspection apparatus. [Figure 2] It is a flowchart for explaining the main process of inspecting the inspection object. [Figure 3] It is a flowchart for explaining the primary determination process executed in the above main process. [Figure 4] This flowchart explains the secondary judgment process performed in the main process described above. [Figure 5] This is a flowchart illustrating the pre-training process in the above-mentioned secondary judgment process. [Figure 6] This is a flowchart illustrating the registration process in the secondary determination process described above. [Figure 7] This is an explanatory diagram for describing the confirmation screen shown to the operator. [Figure 8] This is a flowchart to explain the learning process. [Modes for carrying out the invention]

[0026] Hereinafter, preferred embodiments of the present invention will be described with reference to the drawings. First, the configuration of the inspection apparatus 10 according to this embodiment will be described using Figure 1. This inspection apparatus 10 inspects the object to be inspected 12 using image data (2D image data or pattern image data) of the object to be inspected 12 obtained by imaging the object to be inspected 12. The object to be inspected 12 is, for example, an electronic circuit board (substrate) on which components are mounted and solder is applied.

[0027] The inspection device 10 comprises an inspection table 14 for holding the object to be inspected 12, an imaging unit 20 which is an imaging unit that illuminates and images the object to be inspected 12, an XY stage 16 which moves the imaging unit 20 relative to the inspection table 14, and a control unit 30 which is a control unit that controls the operation of the imaging unit 20 and the XY stage 16 and performs the inspection of the object to be inspected 12. For the sake of explanation, as shown in Figure 1, the surface on which the object to be inspected is placed on the inspection table 14 is defined as the XY plane (orthogonal X and Y directions), and the direction perpendicular to that surface (i.e., the imaging direction by the camera unit 21 that constitutes the imaging unit 20 (the optical axis direction of the optical system of the camera unit 21)) is defined as the Z direction.

[0028] The inspection table 14 is a transport unit, such as a belt conveyor, that brings the object to be inspected 12 into the inspection area within the inspection device 10 and removes the object to be inspected 12 from the inspection area after the inspection is completed, and has the function of a holding unit that holds the object to be inspected 12 when imaging is performed by the imaging unit 20.

[0029] The imaging unit 20 is mounted on a movable table (not shown) of the XY stage 16 and is movable in the X and Y directions by the XY stage 16. The XY stage 16 functions as a drive unit that moves the imaging unit 20 relative to the object under inspection 12, and is, for example, a so-called H-shaped XY stage. Therefore, the XY stage 16 comprises a Y-axis unit consisting of a Y-direction guide extending in the Y direction and a Y-drive unit that moves the movable table in the Y direction along this Y-direction guide, and an X-axis unit consisting of two X-direction guides that support the Y-direction guide at both ends and an X-drive unit that moves the Y-direction guide in the X direction together with the movable table along this X-direction guide. The XY stage 16 also further comprises a Z-axis unit having a Z-drive unit that moves the imaging unit 20 in the Z direction. Furthermore, it may also be equipped with a rotation mechanism that rotates the imaging unit 20 around the optical axis of the optical system of the camera unit 21. The inspection device 10 may further include an XY stage that allows the inspection table 14 to move, in which case the XY stage 16 for moving the imaging unit 20 may be omitted. Linear motors or ball screws can be used for the X drive unit, Y drive unit, and Z drive unit. Alternatively, instead of moving the imaging unit 20 in the Z direction, the XY stage 16 may be configured to move in the Z direction.

[0030] The imaging unit 20 includes a camera unit 21 that captures images from a direction perpendicular to the inspection surface (substrate surface) of the object to be inspected 12 (Z direction), an illumination unit 22, and a projection unit 23. In the inspection apparatus 10 according to this embodiment, the camera unit 21, the illumination unit 22, and the projection unit 23 may be configured as an integrated imaging unit 20. In this integrated imaging unit 20, the relative positions of the camera unit 21, the illumination unit 22, and the projection unit 23 may be fixed, or each unit may be configured to be relatively movable. Alternatively, the camera unit 21, the illumination unit 22, and the projection unit 23 may be separate components and configured to be independently movable.

[0031] The camera unit 21 includes an image sensor (imaging unit) that generates a two-dimensional image of the object, and an optical system (imaging unit consisting of, for example, a lens) for forming an image on the image sensor. This camera unit 21 is, for example, a CCD camera. The maximum field of view of the camera unit 21 (the imaging area with the maximum field of view is called the FOV (Field Of View)) may be smaller than the area on the inspection table 14 where the object to be inspected is placed. In this case, the camera unit 21 images the entire object to be inspected 12 by dividing it into multiple partial images. The control unit 30 controls the XY stage 16 so that the camera unit 21 moves to the next imaging position each time the camera unit 21 captures a partial image. The control unit 30 combines the partial images to generate a whole image of the object to be inspected 12.

[0032] The camera unit 21 may also be equipped with an image sensor that generates a one-dimensional image instead of a two-dimensional image sensor. In this case, the entire image of the object to be inspected 12 can be obtained by scanning the object to be inspected 12 with the camera unit 21. In addition, the imaging unit 20 may be equipped with multiple camera units that image the object to be inspected 12 at angles different from those of the camera unit 21. By imaging the object to be inspected 12 at angles different from those of the camera unit 21, areas that are obscured by parts attached to the object to be inspected 12 and cannot be imaged by the camera unit 21 can be inspected using image data captured at angles different from those of the camera unit 21.

[0033] The illumination unit 22 is configured to project illumination light for imaging by the camera unit 21 onto the surface of the object under inspection 12. The illumination unit 22 includes one or more light sources that emit light of a wavelength or wavelength range selected from the wavelength range detectable by the image sensor of the camera unit 21. The illumination light is not limited to visible light, and ultraviolet light, X-rays, etc., may be used. If multiple light sources are provided, each light source is configured to project light of different wavelengths (e.g., red, blue, and green) onto the surface of the object under inspection 12 at different projection angles.

[0034] The object under inspection 12, illuminated by the lighting unit 22, is imaged by the camera unit 21. The inspection device 10 determines the presence or absence of defects on the substrate of the object under inspection 12 (for example, whether components are present and properly placed, and whether the solder application is good or bad) based on the image data of the object under inspection 12 obtained by being illuminated and imaged by the lighting unit 22 (this image data is called "2D image data") and a height map described later.

[0035] In the inspection apparatus 10 according to this embodiment, the illumination unit 22 is a lateral illumination source that projects illumination light onto the inspection surface of the object to be inspected 12 from an oblique direction, and in this embodiment, it comprises an upper light source 22a, an intermediate light source 22b, and a lower light source 22c. In the inspection apparatus 10 according to this embodiment, the lateral illumination sources 22a, 22b, and 22c are each ring illumination sources, surrounding the optical axis of the camera unit 21 and configured to project illumination light obliquely onto the inspection surface of the object to be inspected 12. Each of these lateral illumination sources 22a, 22b, and 22c may be configured with multiple light sources arranged in a ring. Furthermore, the upper light source 22a, intermediate light source 22b, and lower light source 22c, which are lateral illumination sources, are each configured to project illumination light onto the inspection surface at different angles.

[0036] The projection unit 23 projects a pattern (for example, illumination light with varying light intensity) onto the inspection surface of the object under inspection 12. The object under inspection 12, onto which the pattern is projected, is imaged by the camera unit 21. The inspection device 10 creates a height map of the inspection surface of the object under inspection based on the image data of the object under inspection 12 obtained by imaging (this image data is called "pattern image data"). Here, the height map is data that contains height information of the object under inspection 12 for each pixel of the pattern image data. The control unit 30 detects local discrepancies in the pattern image data with respect to the projection pattern and acquires height information for that part based on the local discrepancy. In other words, changes in the image pattern with respect to the projection pattern correspond to changes in height on the inspection surface.

[0037] The projection pattern is preferably a striped pattern in which different brightness levels with continuity are periodically repeated. The projection unit 23 is positioned to project the striped pattern onto the inspection surface of the object under inspection 12 from an oblique direction. Discontinuities in height on the inspection surface of the object under inspection 12 are represented as pattern shifts in the striped pattern image. Therefore, the height difference can be determined from the amount of pattern shift. For example, the control unit 30 creates a height map using the PMP (Phase Measurement Profilometry) method, which uses a striped pattern whose brightness changes according to a sine curve. In the PMP method, the amount of shift in the striped pattern corresponds to the phase difference of the sine curve.

[0038] The projection unit 23 comprises a pattern forming apparatus, a light source apparatus for illuminating the pattern forming apparatus, and an optical system for projecting the pattern (light transmitted through the pattern forming apparatus) onto the inspection surface of the object to be inspected 12. The pattern forming apparatus may be a variable patterning apparatus capable of dynamically generating a desired pattern, such as a liquid crystal display, or a fixed patterning apparatus in which a pattern is fixedly formed on a substrate such as a glass plate. If the pattern forming apparatus is a fixed patterning apparatus, it is preferable to make the projection position of the pattern variable by providing a moving mechanism for moving the fixed patterning apparatus or by providing an adjustment mechanism in the optical system for pattern projection. Furthermore, the projection unit 23 may be configured to allow switching between multiple fixed patterning apparatuses having different patterns.

[0039] Multiple projection units 23 may be provided around the camera unit 21. The multiple projection units 23 are arranged to project patterns onto the object under inspection 12 from different projection directions. In this way, the area where the pattern is not projected due to shadows caused by height differences on the inspection surface can be reduced.

[0040] The control unit 30 shown in Figure 1 comprehensively controls the entire device. It is implemented as hardware using the CPU, memory, and other LSIs of any computer, and as software using programs loaded into memory. Here, however, the functional blocks realized through the coordination of these components are depicted. Therefore, it will be understood by those skilled in the art that these functional blocks can be realized in various ways using hardware alone, software alone, or a combination thereof.

[0041] Figure 1 shows an example of the configuration of the control unit 30. The control unit 30 includes an inspection control unit 31 and a memory 35 which is a storage unit. The inspection control unit 31 includes an image processing unit 32, an inspection information processing unit 33, and an inspection unit 34. Furthermore, the image processing unit 32 includes an imaging processing unit 32a and a height measuring unit 32b. The inspection device 10 also includes an input unit 36 ​​for receiving input from a user or other device, and an output unit 37 for outputting information related to the inspection. The input unit 36 ​​and the output unit 37 are each connected to the control unit 30. The input unit 36 ​​includes, for example, input means such as a mouse or keyboard for receiving input from a user, and communication means for communicating with other devices. The output unit 37 includes known output means such as a display or printer. Alternatively, the input unit 36 ​​and the output unit 37 may be configured as a touch panel display that integrates these functions.

[0042] As a preprocessing step for creating a height map, the inspection control unit 31 controls the relative movement between the imaging unit 20 and the inspection table 14 while projecting a pattern onto the object under inspection 12 from the projection unit 23 using the imaging processing unit 32a of the image processing unit 32, thereby dividing the pattern image of the object under inspection 12 and capturing it sequentially. The projected pattern is preferably a striped pattern whose brightness changes according to a sine curve based on the PMP method. The inspection control unit 31 synthesizes the captured divided images to generate pattern image data for the entire inspection surface of the object under inspection 12. The inspection control unit 31 stores the pattern image data in the memory 35. Note that pattern image data may be generated for only a part of the inspection surface of the object under inspection 12, rather than the entire surface.

[0043] The height measuring unit 32b creates a height map of the entire inspection surface of the object under inspection 12 based on the pattern image data. First, the height measuring unit 32b obtains a phase difference map of the inspection surface of the object under inspection 12 by determining the local phase difference between the pattern image data and the reference pattern image data for the entire image data. Here, "reference pattern image data" refers to image data in which a pattern is projected onto a reference plane by the projection unit 23 (i.e., image data acquired by the camera unit 21 in which the pattern generated by the pattern forming device built into the projection unit 23 is projected onto the reference plane). The height measuring unit 32b creates a height map of the object under inspection 12 based on the reference plane, which serves as the basis for height measurement, and the phase difference map. The reference plane is, for example, the substrate surface of the electronic circuit board being inspected. The reference plane does not necessarily have to be a plane; it may be a curved surface that reflects deformation such as warping of the substrate.

[0044] Specifically, the height measuring unit 32b determines the phase difference of the stripe pattern between each pixel of the pattern image data and the corresponding pixel of the reference pattern image data. The height measuring unit 32b converts the phase difference into height information. This is because, even if the stripe width of the reference pattern is constant, the stripe width changes from one end to the other of the pattern projection area on the inspection surface because the distance from the projection unit 23 differs depending on the position on the inspection surface. Based on the converted height information and the reference plane, the height measuring unit 32b obtains height information from the reference plane and creates a height map of the object under inspection 12.

[0045] The image processing unit 32 of the inspection control unit 31 may create an image of the object under inspection with a height distribution by associating the height information contained in the height map of the object under inspection 12 with each pixel of the two-dimensional image data of the object under inspection 12. Alternatively, the image processing unit 32 may perform a three-dimensional modeling display of the object under inspection 12 based on the image data of the object under inspection with a height distribution. Furthermore, the image processing unit 32 may overlay the height distribution onto the two-dimensional image data of the object under inspection 12 and display it on the output unit 37. For example, the two-dimensional image data may be color-coded according to the height distribution.

[0046] Furthermore, the inspection control unit 31 is configured to execute various control processes for inspection based on input from the input unit 36 ​​and inspection-related information stored in the memory 35. The inspection-related information includes two-dimensional image data of the object to be inspected 12, a height map of the object to be inspected 12 (calculated from pattern image data as described above), and inspection information such as substrate inspection information and information obtained through learning (hereinafter referred to as "learning result information," which corresponds to, for example, an inference model in the case of an AI-based method). The inspection unit 34 performs the inspection based on the created substrate inspection information and learning result information, and the two-dimensional image data and height map of the object to be inspected 12.

[0047] The circuit board inspection information is inspection information created for each type of circuit board and used in the primary judgment process (rule-based inspection). The circuit board inspection information is a collection of inspection information for each component placed on the circuit board, its position, and the solder applied to the circuit board. The inspection information for each component and solder includes the inspection items required for that component or solder, the inspection window which is the inspection area on the image for each inspection item, and the inspection criteria which serve as the basis for placement and quality judgment for each inspection item. One or more inspection windows are set for each inspection item. For example, in an inspection item that determines the quality of the solder application state, the same number of inspection windows as the number of solder application areas of that component are usually set in an arrangement corresponding to the arrangement of the solder application areas. Furthermore, for inspection items that use image data that has undergone predetermined image processing on the 2D image data of the object under inspection 12, the content of that image processing is also included in the inspection information.

[0048] Here, if multiple inspection rules are applied to a single inspection target (for example, each pin of a semiconductor chip), one inspection window is set for that inspection target, one for each inspection rule. For example, if four inspection rules—"position shift inspection," "floating inspection," "polarity inspection," and "solder inspection"—are applied to the tip of each pin of a semiconductor chip with multiple pins, four inspection windows corresponding to these inspection rules will be set.

[0049] The inspection information processing unit 33 sets each item of the inspection information according to the board as part of the board inspection information creation process. For example, the inspection information processing unit 33 automatically sets the position and size of each inspection window for each inspection item to match the solder layout of the board. The inspection information processing unit 33 may also accept user input for some items of the inspection information. For example, the inspection information processing unit 33 may accept tuning of inspection criteria by the user. The inspection criteria may be set using height information.

[0050] The learning result information is created for each component attached to the substrate and is inspection information used in the secondary judgment process (for example, an inspection using AI, hereinafter referred to as the "inspection method using the learning result"). In the inspection apparatus 10 according to this embodiment, the learning result information is obtained by having the operator make a judgment again on an inspection window that was judged as defective in the primary judgment process, and when the judgment result is different (when it was judged as defective in the primary judgment process but judged as good by the operator), the image data used for that judgment is grouped and stored, and the result is learned, and is generated by the inspection information processing unit 33. Therefore, the inspection information processing unit 33 has the function of a pre-processing unit that groups and stores image data as the operator's judgment result, and the function of a learning unit that learns the operator's judgment result.

[0051] The inspection control unit 31 performs imaging of the object to be inspected 12 using the image processing unit 32 as a preprocessing step for creating substrate inspection information. Here, the image data (2D image data and pattern image data, which are full-surface substrate image data) used for creating the substrate inspection information is from an object to be inspected 12 that has passed all inspection items. As described above, the imaging process is performed by illuminating the object to be inspected 12 with the illumination unit 22, controlling the relative movement of the imaging unit 20 and the inspection table 14, and sequentially capturing partial images of the object to be inspected 12. Multiple partial images are captured so that the entire object to be inspected 12 is covered. The inspection control unit 31 combines these multiple partial images to generate full-surface substrate image data (2D image data) that includes the entire inspection surface of the object to be inspected 12. The inspection control unit 31 stores the full-surface substrate image data in the memory 35.

[0052] The inspection unit 34 of the control unit 30 of the inspection apparatus 10 according to this embodiment is configured to perform an inspection (primary judgment process) for each inspection window of the object to be inspected 12, using two-dimensional image data and a height map (image data of the currently selected inspection window, hereinafter collectively referred to as "inspection target image data") based on the above-described substrate inspection information. For inspection target image data that is judged as "defective" in this primary judgment process, a re-judgment process is performed again, which combines an inspection using the results learned from the operator's visual judgment (secondary judgment process, which is an inspection using the learned results) and an inspection by the operator's visual inspection (visual judgment process). For example, even with the same part, the surface markings (engraved model number, etc.) may differ depending on the manufacturing period and manufacturer. Even if it is judged as defective in the primary judgment process, by performing a re-judgment in the secondary judgment process, it is possible to prevent something that should be judged as "good" from being judged as "defective" (over-judgment). Thus, the inspection unit 34 has the functions of a primary judgment unit that performs the primary judgment process and a re-judgment unit that performs the re-judgment process.

[0053] Here, as the secondary judgment process, the "inspection method using the learning results" can be, for example, the "similar image judgment method," "average image / variance image judgment method," and "anomaly detection judgment method using machine learning" shown below. As described above, all of these methods learn the operator's judgment results (the result when the operator judges an inspection window that was judged as defective in the primary judgment process as good), and perform the secondary judgment process using the learned results (learning result information). The similarity between the image data that is the target of the secondary judgment process (image data of the currently selected inspection window, which is the "inspection target image data" mentioned above) and the learned result information and the learned image data (hereinafter referred to as "learned image data") is calculated, and it is determined that the learned image data with the greatest similarity among the learned image data with a similarity of a threshold or higher is identical, and if there is identical learned image data, it is judged as good.

[0054] In this embodiment, the inspection device 10, when an inspection target image data is determined to be good by the operator during the initial judgment process, is configured to temporarily store the operator's judgment result as temporary registration data and to learn the data when predetermined conditions are met. The image data to be learned is moved from the temporary registration data to the permanent registration data. At this time, the inspection target image data newly added as temporary registration data is compared with the already temporarily registered image data (calculating the similarity described above), and if it is similar to the already temporarily registered image data, the image data is grouped together. This is because learning similar image data has little effect and has little impact on the result of the secondary judgment process. The number of image data in a group is used as a criterion for deciding whether or not to learn the data. The image data representing each group of temporary registration data can be the average image data within the group (average image data), or the first or last registered image data, and learning is performed using the representative image data on a group basis (as described above, groups that meet predetermined conditions are used as the target for learning). Details will be described later.

[0055] Similarity can be expressed, for example, as the degree to which the image data being compared matches the reference image data. A score of 100 indicates that all pixels match, while a score of 0 indicates that all pixels are different.

[0056] The similar image determination method involves learning image data that has been judged as good by an operator's visual inspection and storing it as a template. The method then determines whether an image is good or bad by comparing the image data to be inspected with this template using template matching. Specifically, for image data to be inspected that has been judged as bad in the initial judgment process, the similarity is calculated between each of the image data registered in the template (the learning result information described above). If there is a template (image data) whose similarity is above a predetermined threshold, the image data to be inspected is determined to be identical to the image data of the template with the highest similarity among them, and is judged as "good". In the similar image determination method, when learning from provisional registration data, representative image data of groups that meet predetermined conditions are registered in the template. Alternatively, the phase-only correlation method (a method of matching images using the spectrum after the Fourier transform) may be used instead of template matching.

[0057] The average image / dispersed image determination method learns image data that has been judged as good by the operator's visual inspection. At this time, the average image data and dispersed image data of the image data within the aforementioned group are stored as learning result information. For inspection target data that has been judged as bad in the initial judgment process, the similarity is calculated using the average image data and dispersed image data. Among the groups whose similarity is above a predetermined threshold, the system is configured to determine that the inspection target image data is the same as the group with the highest similarity and judge it as "good". In the average image / dispersed image determination method, when learning from provisionally registered data, the average image data and dispersed image data of groups that meet predetermined conditions are registered.

[0058] When M image data Vk(i,j) are registered as provisional registration data for a certain group, the mean image data Va(i,j) and the variance image data Vσ(i,j) are expressed as shown in the following equations (1) and (2). Here, i indicates the position of a pixel in the image data in the X direction, and j indicates the position of a pixel in the image data in the Y direction. That is, Vk(i,j), Va(i,j), and Vσ(i,j) indicate the value of the pixel at position (i,j). Also, k indicates the identification number within the group of registered image data.

[0059] Va(i,j)=(ΣVk(i,j)) / M (1) Vσ(i,j)=(Σ(Vk(i,j)-Va(i,j)) 2 ) / M (2) However, Σ is the sum up to k=1, ..., M.

[0060] In the mean-image / variance-image determination method, similarity is determined by identifying pixels in the image data under inspection whose values ​​exceed the range of variation of each pixel in the image data within the matching group (these are called "anomalous pixels"), and then determining the similarity based on the proportion of anomalous pixels to the total number of pixels in the image data. Specifically, if the pixel values ​​of the image data under inspection are W(i,j), then from equations (1) and (2) above, an anomalous pixel is a pixel that satisfies the following equation (3). Here, equation (3) defines an anomalous pixel as one whose difference between each pixel in the image data under inspection and the mean value of each pixel in the image data included in the similar group is α times or more the standard deviation (variance) of each pixel in the group's image data. The coefficient α is predetermined.

[0061] W(i,j)-Va(i,j) > Vσ(i,j)×α (3) However, α is a coefficient.

[0062] In the mean-image / dispersion-image classification method, the similarity score can be expressed as 0 when all pixels in the image data being inspected are abnormal pixels, and as 100 when all pixels are not abnormal pixels.

[0063] The machine learning-based anomaly detection method generates an inference model by training on image data of inspected items that were judged as good by an operator's visual inspection, treating them as image data of good items. This inference model is then used to determine whether an item is good or bad. The inference model is applied to the inspected item image data that was judged as bad in the initial judgment process, and the AI ​​is used to perform the inspection. There are two methods for generating the inference model: one for each group as described above, and another for generating a single inference model regardless of the group. In the method of generating an inference model for each group, the inference model for each group is sequentially applied to the inspected item image data that was judged as bad in the initial judgment process to calculate the similarity. Among the groups whose similarity exceeds a predetermined threshold, the group with the highest similarity is determined to be the same as the inspected item image data, and it is judged as "good". On the other hand, in the method of generating a single inference model, the inference model is applied to the inspected item image data that was judged as bad in the initial judgment process, and if the similarity is determined to be above a predetermined threshold, it is judged as "good". When learning from provisional registration data, representative image data of groups that meet predetermined conditions is used for training.

[0064] In the inspection apparatus 10 according to this embodiment, the method used in the secondary determination process may be one of the methods described above, pre-implemented in the control unit 30, or several methods may be implemented and the operator may be allowed to select one via the input unit 36 ​​before the start of the inspection.

[0065] Next, the inspection process of the object to be inspected 12 by the inspection apparatus 10 according to this embodiment will be explained using Figures 2 to 7. As shown in Figure 2, when inspection is started, the inspection control unit 31 of the control unit 30 executes the main process and loads the substrate, which is the object to be inspected 12, into the inspection area on the inspection table 14 of the inspection apparatus 10 (step S100). Then, the inspection control unit 31 uses the imaging processing unit 32a of the image processing unit 32 to image the substrate, which is the object to be inspected 12, that has been loaded into the inspection area, and acquires two-dimensional image data and pattern image data of the object to be inspected 12 (step S102). The acquired two-dimensional image data and pattern image data are stored in the memory 35. In addition, the inspection control unit 31 uses the height measurement unit 32b of the image processing unit 32 to calculate height information using the acquired pattern image data and create a height map (step S104). This height map is also stored in the memory 35.

[0066] Next, the inspection control unit 31 performs an inspection of the object to be inspected 12 (primary determination process) using the two-dimensional image data and height map acquired by the inspection unit 34 (step S106).

[0067] As shown in Figure 3, when the primary determination process S106 is started, the inspection unit 34 selects one of the inspection windows set for the object to be inspected 12 (step S1060), reads the inspection information set for the currently selected inspection window, i.e., the substrate inspection information, from the memory 35, and performs an inspection (primary determination process) using a rule-based inspection method on the 2D image data and height map in the currently selected inspection window based on this substrate inspection information (step S1061), and makes a good or bad determination based on the inspection result (step S1062). If the determination result is judged to be good (step S1062:Y), the inspection unit 34 stores the inspection result of the currently selected inspection window as good in the memory 35 (step S1063). On the other hand, if the determination result is judged to be bad (step S1062:N), the inspection unit 34 stores the inspection result of the currently selected inspection window as bad in the memory 35 (step S1064). The inspection unit 34 then determines whether all inspection windows have been selected (step S1065). If it determines that there are still unselected inspection windows remaining (step S1065:N), it returns to step S1060 to select the next inspection window and repeats the subsequent processing. On the other hand, if it determines that all inspection windows have been selected (step S1065:Y), it terminates the primary determination process S106.

[0068] Returning to Figure 2, the inspection unit 34 determines in the primary determination process S106 whether the inspection results for all inspection windows have been determined to be good or not (step S108). If it is determined that the inspection results for all inspection windows are good (step S108:Y), the inspection unit 34 determines that the object being inspected 12 is a good product and stores it in the memory 35 (step S120). On the other hand, if it is determined that at least one inspection window has been determined to be defective (step S108:N), the inspection unit 34 extracts the inspection window that was determined to be defective from the memory 35 (step S110) and performs a secondary determination process on this extracted inspection window (the inspection window that was determined to be defective). Specifically, the inspection unit 34 selects one inspection window from the inspection windows extracted in step S110 (search windows that were judged to be defective in the primary judgment process) (step S112), and performs a re-judgment process using the learning result information on the selected inspection window (a combination of a secondary judgment process, which is an inspection method using the learning result, and a visual judgment process by an operator) (step S114).

[0069] As shown in Figure 4, the inspection unit 34 calculates the similarity between the image data of the currently selected inspection window and the registered image data, and then performs a secondary judgment process using the learning result information (inspection method using the learning result) (step S1141), and determines whether the judgment result from the secondary judgment process for the currently selected inspection window is poor or not (step S1142).

[0070] The inspection method using the learning results is assumed to be one of the methods described above (similar image determination method, mean image / variance image determination method, anomaly detection determination method using machine learning, etc.), which is pre-configured. Furthermore, if sufficient learning result information has not been accumulated and the inspection method using the learning results cannot be executed, the secondary determination process in step S1141 is not executed. Whether or not to execute the secondary determination process in step S1141 may be determined by the inspection unit 34 according to the accumulated learning result information, or it may be configured so that an operator or the like sets it to the control unit 30. If the secondary determination process is not executed in step S1141, the determination result of the secondary determination process for the currently selected inspection window is assumed to be good, and this inspection window will be used as the target of learning in the process described later. At this time, the operation mode determined in step S1145 described later is assumed to be the "provisional operation mode" which displays a confirmation screen and accepts visual judgment by the operator.

[0071] If the secondary judgment process for the currently selected inspection window determines that it is defective (step S1142:Y), that is, if the result of the secondary judgment process is determined to be the same as the primary judgment process, it is possible that an inspection target (inspection window) that should have been judged as good is being judged as defective in the secondary judgment process (good inspection targets are being overlooked). Therefore, the inspection unit 34 notifies the operator by displaying the location of the abnormality in the inspection target image data on the output unit 37 (step S1144). At this time, the inspection control unit 31 may be configured to be notified in advance via the input unit 36, etc., whether or not to notify the warning in step S1144, and the inspection unit 34 may be configured to determine whether or not to display the warning (step S1143). If it is determined that the setting to display the warning is in place (step S1143:Y), the inspection unit 34 displays the warning (step S1144) and proceeds to step S1146. On the other hand, if it is determined that the system is set not to display a warning (step S1143:N), the inspection unit 34 does not display a warning and proceeds to step S1146.

[0072] The inspection unit 34 displays a confirmation screen including the results of the primary and secondary judgment processes on the output unit 37 to prompt the operator to make a decision (step S1146). Here, the inspection unit 34 displays the image data determined in the primary and secondary judgment processes (the image data of the inspection target, which is the image data of the currently selected inspection window) and the image data of a good product set in the substrate inspection information of the inspection window, as well as an area for the operator to input their decision. The inspection unit 34 then receives the operator's decision (the operator's judgment result from the visual judgment process) via the input unit 36 ​​(step S1147) and determines whether the operator's judgment result is defective or not (step S1148).

[0073] On the other hand, if the secondary judgment process determines that the currently selected inspection window is good (step S1142:N), that is, if the secondary judgment process determines it to be good, which differs from the primary judgment process, there remains a possibility that the secondary judgment process has mistakenly determined an inspection target (inspection window) that should be judged as defective to be good. Therefore, the inspection unit 34 determines whether the current operating mode is "main operating mode" or "provisional operating mode" (step S1145). Here, "operating mode" is a mode for determining whether or not to require operator confirmation (visual judgment process) when the currently selected inspection window is judged to be good as a result of the secondary judgment process. As will be described later, if the secondary judgment process is deemed reliable as a result of learning the operator's judgment, by selecting the main operating mode, it is possible to configure the system so that operator confirmation is not performed when the secondary judgment process determines it to be good, thereby reducing the burden on the operator to deal with over-judgments. On the other hand, if the results of the secondary judgment process are unreliable, such as due to insufficient learning, selecting the provisional operation mode allows the operator to reconfirm the "good" judgment result from the secondary judgment process, thereby preventing over-judgment and preventing inspection windows that should be judged as defective from being judged as good. The operation mode is pre-set in the control unit 30 via the input unit 36.

[0074] If the current operating mode is determined to be not the "main operating mode," i.e., the "temporary operating mode" (step S1145:N), the inspection unit 34 displays the inspection target image data on the output unit 37 along with the results of the primary and secondary judgment processes as described above, requests the operator to visually confirm the result again (step S1146), accepts the operator's judgment (step S1147), and determines whether the operator's judgment result is defective or not (step S1148).

[0075] Furthermore, when displaying the confirmation screen in step S1146, for example, if the average image / dispersion image determination method described above is used as the secondary determination process, the system may be configured to display pixels (abnormal pixels) that satisfy the above-described condition (3) on the image data to be inspected (for example, by changing the color of the abnormal pixels) and present them to the operator.

[0076] When the operator determines that the judgment result is poor (step S1148:Y), the inspection unit 34 determines whether the result of the secondary judgment process performed in step S1141 is good or not (step S1149). If the result of the secondary judgment process is good, but the operator's judgment is poor (step S1149:Y), the secondary judgment process has missed an inspection target (inspection window) that should have been judged as poor. Therefore, the inspection unit 34 stores the image data of the currently selected inspection window as a missed image in memory 35, along with the information of the object being inspected and the information of the currently selected inspection window (step S1150), and terminates the secondary judgment process S114. On the other hand, if the result of the secondary judgment process is determined to be poor (step S1149:N), the result of the secondary judgment process matches the operator's judgment result, so no action is required, and the inspection unit 34 terminates the secondary judgment process S114.

[0077] On the other hand, if the operator's judgment result is determined to be good, which differs from the secondary judgment process (step S1148:N), or if the current operating mode is determined to be the main operating mode (step S1145:Y), the inspection unit 34 stores the inspection result of the currently selected inspection window as good in the memory 35 (step S1151), and proceeds to the pre-learning processing of the operator's judgment result by the inspection information processing unit 33.

[0078] As shown in Figure 5, the inspection information processing unit 33, as a pre-training process, first calculates and compares the similarity between the image data to be inspected in the current inspection window and each of the trained image data in the registered data (a set of trained image data on which training has been performed) (step S1152), and determines whether the image data to be inspected matches any of the registered data (whether the similarity to any of the registered image data is above a predetermined threshold) (step S1153). If it is determined that the image data to be inspected matches any of the registered image data (step S1153: Y), the pre-training process by the inspection information processing unit 33 is terminated because the image data to be inspected has already been trained, and the process returns to Figure 4, terminating the secondary determination process S114 for the currently selected inspection window.

[0079] On the other hand, if it is determined that the image data to be inspected does not match any of the registered image data (step S1153:N), the inspection information processing unit 33 calculates and compares the similarity with each of the provisionally registered data (step S1154) and determines whether the image data to be inspected matches any of the provisionally registered data (whether the similarity with any of the provisionally registered image data is above a predetermined threshold) (step S1155). If it is determined that the image data does not match any of the provisionally registered data (step S1155:N), the inspection information processing unit 33 creates a new group of provisionally registered data, registers the image data to be inspected in the current inspection window to this group, sets the number of image data in the group to 1 (step S1159), terminates the pre-learning processing by the inspection information processing unit 33, returns to Figure 4, and terminates the secondary determination processing S114 for the currently selected inspection window.

[0080] If it is determined that the image data matches any of the provisionally registered data (step S1155:Y), the inspection information processing unit 33 registers the image data to be inspected in the current inspection window to the group to which the matching image data belongs (the group to which the image data with the highest similarity among image data with similarity equal to or greater than a predetermined threshold belongs), and increases the number of image data in that group by 1 (step S1156). Furthermore, if the average image / dispersed image determination method described above is used as the secondary determination process, the inspection information processing unit 33 calculates the average image data and dispersed image data of the image data belonging to the group to which the image data was registered in step S1156 using the above-described formulas (1) and (2), and stores them in memory 35 as the average image data and dispersed image data of that group, or updates them if they are already stored (step S1157). If the secondary determination process is performed using a method different from the average image / dispersed image determination method, this step S1157 may be omitted.

[0081] Then, the inspection information processing unit 33 determines whether the number of image data images temporarily registered in the group where the image data was registered in step S1156 is equal to or greater than a predetermined threshold (step S1158). Here, the predetermined threshold is set in advance to a value of approximately 3 to 10 images. If it is determined that the number of image data images temporarily registered in the currently selected group is less than the predetermined threshold (step S1158:N), the inspection information processing unit 33 terminates the pre-learning processing, returns to Figure 4, and terminates the secondary determination processing S114 for the currently selected inspection window.

[0082] On the other hand, if the system determines that the number of image data images provisionally registered in the currently selected group is equal to or greater than a predetermined threshold (step S1158:Y), the inspection information processing unit 33 executes the final registration process as a post-learning process (step S210).

[0083] As shown in Figure 6, when the main registration process S210 is started, the inspection information processing unit 33 displays a confirmation screen on the output unit 37 prompting the operator to decide whether or not to train the image data of the group (step S2100). Figure 7 shows an example of the confirmation screen 90. The inspection information processing unit 33 displays the representative image data 91 of the currently selected group (image data of candidates to be trained, hereinafter referred to as "main registration recommended image data"), and also displays a group of verification images 92 as information for the operator to decide whether or not to register this main registration recommended image data 91. The group of verification images 92 may be provisionally registered image data, main registration image data, or image data prepared separately for verification. The group of verification images 92 consists of a pair of verification image data 92a and the similarity (%) 92b between this verification image data 92a and the main registration recommended image data 91. It is desirable that the verification image data 92a displayed in the group of verification images 92 be displayed in descending order of similarity.

[0084] Furthermore, the confirmation screen 90 displays a histogram 93 showing the similarity 92b of the verification image data 92a included in the verification image data group 92, as information for the operator to decide whether or not to register the recommended image data 91. It also displays a bar graph 94 showing the similarity 92b of the verification image data 92a displayed in the verification image data group 92, sorted in descending order. The operator can intuitively decide whether or not to register the recommended image data 91 based on these histograms 93 and bar graphs 94.

[0085] On this confirmation screen 90, when the operator selects an element in the histogram 93 or bar graph 94, the verification image data corresponding to the selected element is displayed (for example, the verification image data 92a corresponding to the selected element in the verification image set 92 is highlighted). With this configuration, the operator can visually confirm the relationship between the similarity and the verification image data (the reliability of the similarity).

[0086] The confirmation screen 90 is equipped with a group of buttons 95, which serve as a means of receiving the operator's decision. This group of buttons 95 consists of a registration button 95a that accepts the operator's decision to register (learn) the recommended image 91, a delete button 95b that accepts the operator's decision not to register (not to learn), and a hold button 95c that accepts the operator's decision to temporarily postpone the decision on whether or not to register.

[0087] Returning to Figure 6, the inspection information processing unit 33 receives instructions from the operator by detecting the operation of the aforementioned button group 95 (step S2101) and makes a judgment on these instructions (step S2102). If it is determined that the operator's instructions are to be rejected (step S2102: rejected), the inspection information processing unit 33 deletes the currently selected group from the provisional registration data or saves it to the backup area of ​​memory 35 (step S2103), returns to Figure 5, and terminates the main registration process S210. If it is determined that the operator's instructions are to be held (step S2102: held), the inspection information processing unit 33 returns to Figure 5 and terminates the main registration process S210. If the operator's judgment is rejected, the image data currently being judged (inspection target image data) is deleted from the provisional registration data group by group. Therefore, the next time similar image data becomes inspection target image data, a new group will be created and provisionally registered. Furthermore, if the operator's decision is pending, the image data currently under consideration (the recommended image data for registration and its group, which are learning candidates) will remain as provisional registration data. The next time similar image data becomes the image data to be inspected, a confirmation screen will be displayed again, prompting the operator to decide whether or not to register it.

[0088] On the other hand, if the operator's instruction is determined to be registration (step S2102: registration), the inspection information processing unit 33 moves the representative image data, average image data, and variance image data of the currently selected group to the area of ​​the registered data in memory 35 and deletes them from the area of ​​the temporary registered data (step S2104). Then, the inspection information processing unit 33 executes a learning process using the image data moved to the registered data according to the inspection method using the learning results set as the secondary judgment process (in the example above, similar image judgment method, average image / variance image judgment method, anomaly detection judgment method using machine learning, etc.) (step S2105), returns to Figure 5, and terminates the registered data process S210. The learning method is as described above.

[0089] Thus, in the inspection device 10 according to this embodiment, in the post-learning processing (main registration processing), a step is provided to approve the judgment result of the image data to be learned by filtering whether or not to learn based on the result of the operator's judgment (register, reject, hold) on the image data to be learned. In this way, the reliability of the judgment method using the learning result can be improved by involving the operator in the selection of image data to be learned.

[0090] Furthermore, the operator (inspector) who visually inspects the inspection windows that have been determined to be defective in the primary and secondary judgment processes may be separated from the operator (administrator) who decides whether or not to proceed with the main registration process. In this case, an output unit 37 for the inspector and an output unit 37 for the administrator may be provided, and the control unit 30 may be configured to switch the output destination depending on the situation (when accepting a judgment from the inspector, a confirmation screen, etc., is displayed on the inspector's output unit 37, and when accepting a decision on whether or not to proceed with the main registration, a confirmation screen, etc., is displayed on the administrator's output unit 37).

[0091] Returning to Figure 5, once the registration process S210 is completed, the pre-learning and post-learning processing by the inspection information processing unit 33 is completed, and then the process returns to Figure 4, completing the secondary determination process S114 for the currently selected inspection window.

[0092] Returning to Figure 2, once the re-evaluation process S114 for the currently selected inspection window is completed, the inspection unit 34 determines whether all inspection windows extracted in step S110 (i.e., all inspection windows that were determined to be defective in the primary evaluation process) have been selected (step S116). If it determines that there are any unselected inspection windows (step S116:N), it returns to step 112 to select the next inspection window and repeats the secondary evaluation process.

[0093] If the inspection unit 34 determines that a re-evaluation process has been performed on all inspection windows that were determined to be defective in the initial evaluation process (step S116:Y), it re-evaluates whether the inspection results for all inspection windows were determined to be good (step S118). If it determines that the inspection results for all inspection windows are good (step S118:Y), the inspection unit 34 determines that the currently inspected object 12 is a good product and stores it in the memory 35 (step S120). On the other hand, if it determines that at least one inspection window was determined to be defective (step S118:N), the inspection unit 34 determines that the currently inspected object 12 is a defective product and stores it in the memory 35 (step S122).

[0094] Finally, the inspection control unit 31 removes the object to be inspected 12 from the inspection area (step S124), completing the inspection for one object to be inspected 12. If there is another object to be inspected 12, the inspection control unit 31 executes the inspection from step S100, repeating the process of loading, imaging, inspection, and loading. Here, after the imaging of the object to be inspected 12 in step S102 is completed, the processing from step S104 onwards is executed, the object to be inspected 12 is removed from the inspection area, and the next object to be inspected 12 is loaded into the inspection area and imaged. This allows the inspection of the previous object to be inspected 12 and the imaging of the next object to be inspected 12 to be performed in parallel, thereby shortening the cycle time required for inspection.

[0095] In the above description, the system is configured to perform a primary judgment process on the object to be inspected 12 that has been brought in, followed by a re-judgment process, and then the object to be inspected 12 is removed. However, it is also possible to configure the system to perform the primary judgment process, then remove the object to be inspected 12, and then perform the re-judgment process. In this case, the re-judgment process may be configured to be performed by a device other than the control unit 30 (such as a computer or other terminal).

[0096] Furthermore, in the explanation above, the pre-learning process was configured to execute the final registration process, which is the post-learning process, when the number of cards provisionally registered in a group exceeds a predetermined number. However, the final registration process may be executed as a separate process (learning process) independent of the inspection process. In this case, steps S1157 and S210 shown in Figure 5 are not executed, and the learning process shown in Figure 8 is executed at a different time (for example, after the end of the day's inspection process).

[0097] As shown in Figure 8, when the learning process starts, the inspection information processing unit 33 selects one group from the provisional registration data (step S200). The inspection information processing unit 33 then determines whether the number of provisionally registered image data in the currently selected group is greater than or equal to a predetermined threshold (step S202). As mentioned above, the predetermined threshold is set to a value of approximately 3 to 10 images. If it is determined that the number of provisionally registered image data in the currently selected group is greater than or equal to the predetermined threshold (step S202:Y), the inspection information processing unit 33 executes the post-learning process, which is the final registration process, as described above (step S210). On the other hand, if it is determined that the number of provisionally registered image data in the currently selected group is less than the predetermined threshold (step S202:N), and when the final registration process S210 is completed, the inspection information processing unit 33 determines whether all groups of provisional registration data have been selected (step S212). If it is determined that there are still groups that have not been selected (step S212:N), the inspection information processing unit 33 returns to step S200 to select the next group and repeats the subsequent processing. On the other hand, if it is determined that all groups have been selected (step S212:Y), the inspection information processing unit 33 terminates the learning process.

[0098] Furthermore, the image data registered as secondary judgment missed images in step S1150 described above can be visually inspected by an operator at the end of the day's inspection process or at any time before the start of full operation to confirm whether or not it is an image that should have been judged as defective (this image is called a "missed image"). If the operator's visual inspection reveals that there are no missed images among the image data registered as secondary judgment missed images, a decision can be made to move from provisional operation to full operation, or if full operation has already been moved, a decision can be made to continue full operation as is. On the other hand, if a missed image is found, the group corresponding to that missed image can be deleted from the registered image data, and the learning process can be run again to allow for further learning.

[0099] Furthermore, the above description is based on an inspection device 10 that implements so-called "automated optical inspection (AOI)," which performs inspection using two-dimensional image data (and height maps generated from pattern image data) obtained by imaging the object to be inspected 12 with a camera unit 21. However, the above inspection method may also be applied to an inspection device that implements so-called "automated X-ray inspection (XOI)," which performs inspection using three-dimensional image data (reconstructed image data or cross-sectional image data) generated from transmission image data obtained by irradiating the object to be inspected 12 with X-rays.

[0100] It should be noted that the above embodiments do not limit the invention described in the claims, and not all combinations of characteristic features described in the embodiments are necessarily essential to the solution. [Explanation of Symbols]

[0101] 10 Inspection equipment 12. Subject under inspection 20. Imaging Unit (Imaging Section) 30 Control Unit (Control Section) 33. Inspection Information Processing Unit (Pre-processing Unit, Learning Unit) 34. Inspection Department (Primary Judgment Department, Re-judgment Department)

Claims

1. An imaging unit that images the object to be examined, The system includes a control unit that performs an inspection of the object to be inspected based on the image data of the object to be inspected captured by the imaging unit, The control unit, A primary determination unit that determines whether the object to be inspected is good or bad using the aforementioned image data, A re-determination unit that uses image data determined to be defective by the primary determination unit to determine whether the inspected object is good or bad in a different way than the primary determination unit, A preprocessing unit that groups and stores image data that has been determined to be defective by the primary determination unit and determined to be good by the re-determination unit, A learning unit learns the state of the object under inspection when it is good, using the image data of the group that satisfies predetermined conditions from the image data grouped by the preprocessing unit, An inspection device having a

2. The re-determination unit is configured to present the image data to the operator and receive the result of the operator's visual inspection. The inspection apparatus according to claim 1.

3. The re-evaluation unit is configured to make a secondary determination of whether the object under inspection is good or bad based on the results learned by the learning unit. The inspection apparatus according to claim 1.

4. When the learning unit learns the image data of the group, it generates and stores average image data, in which the average value of each pixel at the same position in the image data within the group is set as the value of the pixel, and distributed image data, in which the variance value of each pixel at the same position in the image data within the group is set as the value of the pixel. The re-evaluation unit calculates the pixel-by-pixel difference between the image data learned by the learning unit and the image data of the subject of the inspection, using the average image data and the variance image data. The inspection apparatus according to claim 3.

5. The re-evaluation unit, when the result of the secondary evaluation is unsatisfactory, displays the abnormal location on the image data and issues a warning. The inspection apparatus according to claim 3.

6. The re-evaluation unit is configured to present the image data to the operator and accept the result of the operator's visual inspection when the result of the secondary evaluation is unsatisfactory. The preprocessing unit groups and stores the image data when it is determined to be good by the visual inspection. The inspection apparatus according to claim 3.

7. The control unit is configured to be able to switch between a main operation mode and a temporary operation mode as operating modes. The re-evaluation unit is configured to present the image data to the operator and accept the result of the operator's visual evaluation when the result of the secondary evaluation is good and the operation mode is in provisional operation mode. The preprocessing unit groups and stores the image data when it is determined to be good by the visual inspection. The inspection apparatus according to claim 6.

8. The re-evaluation unit stores the image data as missed image data if the result of the visual inspection is poor and the result of the secondary evaluation is good. The inspection apparatus according to claim 6 or 7.

9. When the preprocessing unit stores new image data, The similarity between the newly created image data and the already stored image data is calculated, and image data with a similarity of a predetermined value or higher is grouped together. The inspection apparatus according to claim 1.

10. The preprocessing unit calculates the similarity using template matching or the phase-limit correlation method. The inspection apparatus according to claim 9.

11. The learning unit, as a predetermined condition, learns using the image data of a group when the number of image data belonging to that group exceeds a predetermined number. The inspection apparatus according to claim 1.

12. When the learning unit learns the image data of the group, it learns using the image data of a representative of the group. The inspection apparatus according to claim 1.

13. When the learning unit learns the image data of the group, it stores the image data of the group as learned image data. The inspection apparatus according to claim 1.

14. When the learning unit learns using the image data of the group, it receives a decision from the operator as to whether or not to learn the image data of the group, and when it receives a decision to learn, it executes the learning process. The inspection apparatus according to claim 1.

15. When the learning unit receives the operator's judgment, The similarity of the verification image data to the aforementioned image data used to determine whether or not to train is calculated. The screen used when accepting the request displays the image data used to determine whether or not to perform the learning process, the image data used for verification, and the similarity score. The inspection apparatus according to claim 14.

16. The aforementioned learning unit, When displaying the image data used to determine whether or not to perform learning, the verification image data, and the similarity score on the screen used to perform the aforementioned reception, at least one of the histograms or graphs of the similarity score is displayed. Selecting an element of the histogram or graph will display the verification image data corresponding to that element. The inspection apparatus according to claim 15.

Images