Inspection equipment, component mounting machine, production system, inspection method, method for generating trained models, inspection program, and recording medium.

Abstract

This enables improved accuracy in detecting anomalies present in the target object. [Solution] An inspection can be performed in two ways: a non-AI detection process (first anomaly detection process) on the image Ii to be inspected to detect an anomaly present in the substrate B (object) (step S101); and an AI detection process (second anomaly detection process) different from the non-AI detection process (second anomaly detection process) on the image Ii to be inspected to detect an anomaly present in the substrate B (step S104). Therefore, even if the non-AI detection process (step S101) fails to detect an anomaly, the AI ​​detection process (step S104) can successfully detect the anomaly.

Description

Technical Field

[0001] This invention relates to an inspection technique for detecting abnormalities existing in an object.

Background Art

[0002] Patent Document 1 describes an inspection apparatus that detects abnormalities such as defects existing in an object based on an image acquired by imaging the object. This inspection apparatus inspects an image of the object to detect defects and classifies the detected defects using a learned model.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In a technique for performing a predetermined abnormality detection process on an image to detect an abnormality existing in an object shown in the image, improvement of detection accuracy is important. However, there is a limit to improving the accuracy of the abnormality detection process, and it has been difficult to obtain a desired accuracy.

[0005] This invention has been made in view of the above problems, and an object thereof is to enable improvement in the accuracy of detecting an abnormality existing in an object.

Means for Solving the Problems

[0006] The inspection apparatus according to the present invention includes a camera that images an object to acquire an inspection target image of the object, a first abnormality detection unit that detects an abnormality existing in the object by performing a first abnormality detection process on the inspection target image, and a second abnormality detection unit that detects an abnormality existing in the object by performing a second abnormality detection process different from the first abnormality detection process on the inspection target image.

[0007] The component mounting machine according to the present invention comprises the above-mentioned inspection device and a mounting head for mounting components onto an object.

[0008] The production system according to the present invention comprises the above-mentioned inspection device and a component mounting machine for mounting components onto an object.

[0009] The inspection method according to the present invention comprises the steps of: acquiring an inspection target image of the object by imaging the object with a camera; detecting an abnormality present in the object by performing a first abnormality detection process on the inspection target image; and detecting an abnormality present in the object by performing a second abnormality detection process different from the first abnormality detection process on the inspection target image.

[0010] The inspection program according to the present invention causes a computer to execute the above inspection method.

[0011] The recording medium according to the present invention records the above-mentioned inspection program in a way that can be read by a computer.

[0012] In the present invention configured in this way, an inspection can be performed in which an abnormality present in an object is detected by performing a first abnormality detection process on the image to be inspected, and an inspection can be performed in which an abnormality present in an object is detected by performing a second abnormality detection process different from the first abnormality detection process on the image to be inspected. Therefore, even if the detection of an abnormality fails in the first abnormality detection process, the detection of the abnormality can be successfully performed in the second abnormality detection process. As a result, it becomes possible to improve the accuracy of detecting abnormalities present in an object.

[0013] Furthermore, the inspection device may be configured such that the first anomaly detection process extracts candidate regions from the image to be inspected by performing predetermined image processing, and if the index value of the degree of anomaly in the candidate region is equal to or greater than the anomaly detection threshold, it is determined that an anomaly exists in the candidate region, and the second anomaly detection process detects anomalies that exist in a predetermined limited area narrower than the image to be inspected, including candidate regions with index values ​​below the anomaly detection threshold. In such a configuration, anomalies that could not be detected by the first anomaly detection process because they have index values ​​below the anomaly detection threshold can be detected by the second anomaly detection process. As a result, it becomes possible to improve the accuracy of detecting anomalies present in the object.

[0014] Furthermore, the inspection device may be configured such that, when the first anomaly detection unit extracts candidate regions having an index value below the anomaly detection threshold in the first anomaly detection process, the second anomaly detection unit detects anomalies in a predetermined limited region that is narrower than the image under inspection, including the candidate regions below the anomaly detection threshold extracted by the first anomaly detection process. In such a configuration, the second anomaly detection process can be avoided from being executed unnecessarily, thereby reducing the amount of computation and time required for inspection.

[0015] The specific form of the index value can be various. For example, the index value may be area, or it may be similarity to a given pattern.

[0016] Alternatively, the inspection device may be configured such that candidate regions below the anomaly detection threshold are set as limited regions.

[0017] Furthermore, the inspection device may be configured to include a storage unit for storing a reference image showing a normal object, and the first anomaly detection process extracts candidate regions by performing image processing that compares the image to be inspected with the reference image. This allows the detection of anomalies by the first anomaly detection process to be performed with a certain degree of accuracy.

[0018] Furthermore, the inspection device may be configured such that the second anomaly detection process uses a trained model, which has been trained on a dataset including training images representing the target object and anomaly guidance data indicating the anomalies contained in the training images, to detect anomalies present in the target object. In such a configuration, anomalies that were not detected in the first anomaly detection process can be detected by the trained model. As a result, it becomes possible to improve the accuracy of detecting anomalies present in the target object.

[0019] Furthermore, the inspection device may be configured such that the anomaly training data consists of data annotated with locations where anomalies exist. In such a configuration, anomaly detection using a trained model can be performed with a certain degree of accuracy.

[0020] Furthermore, the inspection device may be configured such that the second anomaly detection unit performs additional training using the image to be inspected and a dataset containing data on anomalies contained in the image to be inspected, thereby updating the existing trained model. In such a configuration, the accuracy of the trained model can be improved through the execution of the inspection.

[0021] The method for generating a pre-trained model according to the present invention is a method for generating a pre-trained model used in an inspection method for detecting anomalies present in an object, and comprises the steps of: acquiring a source image showing the object; acquiring a modified image different from the source image by modifying the source image; and causing the pre-trained model to perform machine learning on a dataset including the modified image and anomaly training data indicating the anomalies contained in the modified image. With this configuration, a large number of anomaly images can be automatically generated and used to train the pre-trained model. Therefore, the accuracy of the pre-trained model can be improved. As a result, the accuracy of detecting anomalies present in the object can be improved. Note that the source image may or may not be used as training data. [Effects of the Invention]

[0022] As described above, the present invention makes it possible to improve the accuracy of detecting abnormalities present in an object.

Brief Description of Drawings

[0023] [Figure 1] Block diagram showing an example of the production system of the present invention. [Figure 2] Partial plan view schematically showing an example of the component mounter included in the production system of FIG. 1. [Figure 3] Block diagram showing an example of the electrical configuration included in the component mounter of FIG. 1. [Figure 4] Block diagram showing the electrical configuration of an example of an inspection device. [Figure 5] Flowchart showing an example of the abnormality inspection executed by the inspection device of FIG. 4. [Figure 6A] Flowchart showing a first example of the non-AI detection process executed in the abnormality inspection of FIG. 5. [Figure 6B] Diagram schematically showing a method of setting a threshold value used in the non-AI detection process of FIG. 6A. [Figure 7A] Flowchart showing an example of the index value acquisition process executed in the first example of the non-AI detection process of FIG. 6A. [Figure 7B] Diagram schematically showing the operations executed in the index value acquisition process of FIG. 7A. [Figure 8A] Diagram schematically showing the operations executed in steps S104 to S106 of the abnormality inspection of FIG. 5. [Figure 8B] Diagram schematically showing the learning content of the learned model used in the AI detection process. [Figure 9A] Flowchart showing a second example of the non-AI detection process executed in the abnormality inspection of FIG. 5. [Figure 9B] Flowchart showing an example of the index value acquisition process executed in the second example of the non-AI detection process of FIG. 9A. [Figure 9C] Diagram schematically showing the operations executed in the index value acquisition process of FIG. 9B. [Figure 9D] Diagram schematically showing an example of the similarity map acquired in the index value acquisition process of FIG. 9B. [Figure 10] Flowchart showing the mounting start control for executing an abnormality inspection in association with the component mounting operation by the component mounter. [Figure 11] A flowchart illustrating the in-process control that performs abnormality checks in relation to the component placement operation by a component placement machine. [Figure 12A] A schematic diagram illustrating how to generate datasets for machine learning. [Figure 12B] A schematic diagram illustrating image modification using annotated images. [Figure 13A] A schematic diagram illustrating the method for generating a dataset for learning solder bridge anomalies. [Figure 13B] A schematic diagram illustrating the method for generating a dataset for learning solder bridge anomalies. [Modes for carrying out the invention]

[0024] Figure 1 is a block diagram showing an example of the production system of the present invention, Figure 2 is a schematic partial plan view showing an example of a component mounting machine included in the production system of Figure 1, and Figure 3 is a block diagram showing an example of the electrical configuration of the component mounting machine of Figure 1. In this embodiment, the horizontal direction X, the horizontal direction Y perpendicular to the X direction, and the vertical direction Z are shown as appropriate. Also, the plan view shows the view from above in the Z direction. The production system 1 produces a component-mounted substrate by mounting components E onto the substrate B. This production system 1 includes a component mounting machine 2 for mounting components E onto the substrate B and an inspection device 5 for inspecting the substrate B.

[0025] As shown in Figure 3, the component mounting machine 2 is equipped with a control unit 29 that comprehensively controls the component mounting machine 2. The control unit 29 includes an arithmetic processing unit 291, a storage unit 292, a drive control unit 293, an imaging control unit 294, a UI 295, and a communication unit 296. The arithmetic processing unit 291 is a processor such as a CPU (Central Processing Unit) that is responsible for the arithmetic functions of the component mounting machine 2, and the storage unit 292 is a storage device such as an SSD (Solid State Drive) or HDD (Hard Disk Drive). The drive control unit 293 controls the drive system provided in the component mounting machine 2 in accordance with commands from the arithmetic processing unit 291, and the imaging control unit 294 controls the imaging system provided in the component mounting machine 2 in accordance with commands from the arithmetic processing unit 291. The UI 295 is a user interface and has input devices such as a mouse or keyboard that accept user input operations, and output devices such as a display that output information to the user. Note that the input and output devices of the UI 295 do not need to be configured separately, and may be configured as an integrated unit by a touch panel display or the like. The communication unit 296 communicates with the inspection device 5 wirelessly or via a wired connection.

[0026] The component mounting machine 2 includes a substrate transport unit 21. The substrate transport unit 21 has a pair of conveyors 211 arranged parallel to the X direction, and the substrate B is transported in the X direction (substrate transport direction) by the pair of conveyors 211. Specifically, the substrate transport unit 21 receives the substrate B from the upstream side in the X direction in response to a loading command from the drive control unit 293 and holds the substrate B at a predetermined substrate holding position Lb (the position of substrate B in Figure 1). The substrate transport unit 21 also transports the substrate B with components E mounted at the substrate holding position Lb downstream in the X direction from the substrate holding position Lb in response to a loading command from the drive control unit 293.

[0027] The component mounting machine 2 is equipped with two mounting heads 22. The mounting heads 22 are in-line type mounting heads having a plurality of mounting shafts 221 arranged in a row in the X direction, with each mounting shaft 221 extending in the Z direction. A suction nozzle for picking up components E is attached to the lower end of each mounting shaft 221, and the mounting shaft 221 holds the components E by the nozzle attached to the mounting shaft 221. The component mounting machine 2 is also equipped with a Z motor 222 attached to the mounting head 22. The Z motor 222 is provided for each of the plurality of mounting shafts 221 of the mounting head 22 and drives the corresponding mounting shaft 221 in the Z direction. In other words, the mounting head 22 can individually raise and lower each of the mounting shafts 221 using the Z motor 222. The mounting head 22 mounts components E to the substrate B by placing the components E held by the nozzles onto the substrate B. Furthermore, the specific type of mounting head 22 is not limited to an inline type; a rotary type with multiple mounting shafts 221 arranged circumferentially is also acceptable.

[0028] Furthermore, the component mounting machine 2 is equipped with an XY drive mechanism 23 that drives each of the two mounting heads 22 individually in the X and Y directions. This XY drive mechanism 23 has two X beams 231, one for each of the two mounting heads 22, and each X beam 231 extends parallel to the X direction and supports the corresponding mounting head 22 so as to be movable in the X direction. A ball screw 232 extending parallel to the X direction and an X motor 233 that rotationally drives the ball screw 232 are attached to the X beam 231. In this example, the X motor 233 is a servo motor. The mounting heads 22 are attached to the nuts of the ball screws 232 on the X beam 231. In addition, the XY drive mechanism 23 has a pair of Y beams 234 that extend parallel to the Y direction. Both ends of each X beam 231 are supported so as to be movable in the Y direction by a pair of Y beams. A Y motor 235 that drives the X beam 231 in the Y direction is attached to each Y beam 234. Each Y motor 235 is a linear motor in this example. With this XY drive mechanism 23, the mounting head 22 can be moved in the X and Y directions by the X motor 233 and the Y motor 235.

[0029] Furthermore, the component mounting machine 2 includes two component supply units 24 located on both sides of the board transport unit 21 in the Y direction. Each component supply unit 24 is detachably fitted with multiple feeders 241 arranged in the X direction, and each feeder 241 supplies components E such as transistors and capacitors to component supply positions Ls. As feeders 241, tape feeders that supply components E to component supply positions Ls by intermittently feeding a tape containing components E in the Y direction, or tray feeders that supply components E to component supply positions Ls by feeding a tray containing components E in the Y direction can be used.

[0030] The mounting head 22 then retrieves the component E supplied by the feeder 241 to the component supply position Ls from the component supply position Ls and places it on the substrate B held at the substrate holding position Lb, thereby performing component mounting. In other words, the drive control unit 293 drives the mounting head 22 with the XY drive mechanism 23 to position the nozzle attached to the lower end of the mounting shaft 221 of the mounting head 22 facing the component E supplied to the component supply position Ls from above. The drive control unit 293 then lowers the mounting shaft 221 with the Z motor 222 to lower the nozzle to the component E and holds the component E with the nozzle. The drive control unit 293 also raises the mounting shaft 221 with the nozzle holding the component E attached using the Z motor 222 to retrieve the component E from the component supply position Ls. Next, the drive control unit 293 drives the mounting head 22 with the XY drive mechanism 23, so that the mounting shaft 221 of the mounting head 22, which is held by the nozzle, faces the mounting point on the substrate B from above. Then, the drive control unit 293 lowers the mounting shaft 221 and the nozzle with the Z motor 222, so that the component E held by the nozzle comes into contact with the surface of the substrate B, and the mounting head 22 releases the nozzle from holding the component E, thus releasing the component E from the nozzle to the substrate B. In this way, the component E is mounted on the substrate B.

[0031] Furthermore, the component mounting machine 2 is equipped with two substrate recognition cameras Cb, which are provided to correspond to the two mounting heads 22. The substrate recognition cameras Cb are fixed to the corresponding mounting heads 22 and move together with the mounting heads 22 in the X and Y directions. These substrate recognition cameras Cb are mounted facing downwards and are used to image, for example, fiducial marks attached to substrate B from above. In addition, the substrate recognition cameras Cb acquire an inspection target image Ii for inspecting substrate B by imaging substrate B. The imaging control unit 294 acquires the inspection target image Ii from the substrate recognition cameras Cb and transmits the inspection target image Ii to the inspection device 5 via the communication unit 296.

[0032] The component mounting machine 2 is equipped with two component recognition cameras Ce, each positioned on either side of the substrate transport unit 21 in the Y direction. The component recognition cameras Ce are positioned facing upwards and are used to image components E held by the nozzle on the mounting shaft 221 from below. In particular, the component recognition cameras Ce image components E before they are taken out of the component supply position Ls by the nozzle and mounted on the substrate B. The component recognition image Ir captured by the component recognition cameras Ce is transmitted to the imaging control unit 294. The imaging control unit 294 then recognizes the orientation of the component E held by the nozzle based on the component recognition image Ir.

[0033] Figure 4 is a block diagram showing the electrical configuration of an example inspection device. The inspection device 5 includes an image processing unit 51, a storage unit 52, a UI 53, and a communication unit 54. The image processing unit 51 is composed of a processor, memory, etc. The storage unit 52 is a storage device such as an SSD or HDD. The UI 53 is a user interface and includes input devices such as a mouse or keyboard that accept user input operations, and output devices such as a display that output information to the user. Note that the input and output devices of the UI 53 do not need to be configured separately and may be integrated using a touch panel display or the like. The communication unit 54 communicates wirelessly or via wired connection with the communication unit 296 of the component mounting machine 2.

[0034] The communication unit 54 stores the inspection target image Ii received from the communication unit 296 of the component mounting machine 2 in the storage unit 52. The storage unit 52 also stores a reference image Im showing a normal circuit board B. Furthermore, the storage unit 52 stores an inspection program Pi that causes the image processing unit 51 to perform an inspection based on the inspection target image Ii. This inspection program Pi is downloaded from an external server computer (recording medium) by the communication unit 54 and stored in the storage unit 52.

[0035] The image processing unit 51 executes the inspection program Pi to configure the non-AI detection unit 61, the AI ​​detection unit 62, the detection area limiting unit 63, and the detection area synthesis unit 64 within the image processing unit 51. The AI ​​detection unit 62 has a trained model 621 that has learned the relationship between the image to be inspected Ii and the abnormalities shown in the image to be inspected Ii.

[0036] Figure 5 is a flowchart showing an example of an anomaly inspection performed by the inspection device in Figure 4; Figure 6A is a flowchart showing a first example of a non-AI detection process performed in the anomaly inspection in Figure 5; Figure 6B is a schematic diagram showing how to set the threshold used in the non-AI detection process in Figure 6A; Figure 7A is a flowchart showing an example of an index value acquisition process performed in the first example of the non-AI detection process in Figure 6A; and Figure 7B is a schematic diagram showing the calculations performed in the index value acquisition process in Figure 7A. The anomaly inspection in Figure 5 is performed by the image processing unit 51 according to the inspection program Pi.

[0037] In step S101 of the anomaly inspection shown in Figure 5, a non-AI detection process is performed. In step S201 of the non-AI detection process shown in Figure 6A, an index value acquisition process is performed to obtain the area of ​​the region where anomalies may exist (candidate region) as an index value. In step S301 of the index value acquisition process shown in Figure 7A, the non-AI detection unit 61 acquires the inspection target image Ii from the storage unit 52. The non-AI detection unit 61 also acquires the reference image Im from the storage unit 52 (step S302). In the example shown in Figure 7B, the reference image Im and the inspection target image Ii each represent a region of the substrate B where a BGA (Ball Grid Array) with multiple ball electrodes arranged is provided. The inspection target image Ii shows multiple chip components scattered on the substrate B. On the other hand, the reference image Im shows a normal substrate B without foreign matter such as chip components.

[0038] In step S303, the non-AI detection unit 61 calculates a difference image Id, which shows the difference between the image to be inspected Ii and the reference image Im. For example, the non-AI detection unit 61 calculates the difference image Id by determining the difference in brightness for each pixel, which is obtained by subtracting the brightness shown in the reference image Im from the brightness shown in the image to be inspected Ii. As a result, as shown in Figure 7B, a difference image Id that highlights multiple chip components scattered on the substrate B can be obtained.

[0039] In step S304, the non-AI detection unit 61 generates a binary image by binarizing the brightness of each pixel in the difference image Id at a predetermined brightness threshold. In step S305, the non-AI detection unit 61 generates candidate regions by connecting adjacent pixels with high brightness in the binary image, which has been binarized into low brightness and high brightness. In step S306, the non-AI detection unit 61 calculates the area of ​​the candidate regions. This area is expressed, for example, in units of pixel area U.

[0040] Once the index value acquisition process (step S201) is completed, step S202 in Figure 6A is executed. In step S202, the count value N, which counts the candidate regions included in the binary image, is reset to zero, and in step S203, the count value N is incremented by 1.

[0041] In step S204, the non-AI detection unit 61 determines whether the area of ​​the Nth candidate region R(N) is greater than or equal to the threshold area Aa. If the area of ​​the candidate region R(N) is greater than or equal to the threshold area Aa (if the answer in step S204 is "YES"), the non-AI detection unit 61 determines that this candidate region R(N) is an abnormal region (step S205). In other words, the candidate region R(N) is determined to be a region where an abnormality such as a foreign object exists.

[0042] This threshold area Aa can be set in various ways, for example, by the method shown in Figure 6B. The chip component in Figure 6B has a length El and a width Ew. If we express the area of ​​the chip component in units of pixel area U, the area of ​​the chip component is given by El × Ew / U. Multiplying this by an appropriate margin coefficient α, we can set the threshold area Aa to α × El × Ew / U. Incidentally, in a binary image, it is conceivable that only the electrodes of the chip component appear at high brightness. In this case, using the length Ee of the electrodes, we can set the threshold area Aa to α × Ee × Ew / U. The margin coefficient α can be set to a value greater than 0 and less than 1.

[0043] If the area of ​​candidate region R(N) is less than the threshold area Aa (if the answer is "NO" in step S204), the process proceeds to step S206. In step S206, the non-AI detection unit 61 determines whether the area of ​​candidate region R(N) is greater than or equal to the threshold area Ab. Here, the threshold area Ab is smaller than the threshold area Aa. If the area of ​​candidate region R(N) is greater than or equal to the threshold area Ab (if the answer is "YES" in step S206), the non-AI detection unit 61 determines that this candidate region R(N) is an uncertain region Ru (step S207).

[0044] Here, the uncertain region Ru indicates a region where it is uncertain whether or not an abnormality exists in the candidate region R(N). For example, in the inspection image Ii in Figure 7B, the chip component enclosed by the white oval overlaps with the ball electrode. Therefore, as shown by the white oval in the difference image Id, the chip component does not appear with a sufficient area. For this reason, the candidate region R representing the chip component is determined to be the uncertain region Ru.

[0045] If the area of ​​candidate region R(N) is less than the threshold area Ab (i.e., "NO" in step S206), the non-AI detection unit 61 determines that candidate region R(N) is a normal region (step S208). In other words, candidate region R(N) is determined to be a region where no abnormalities such as foreign objects exist.

[0046] In step S209, it is determined whether the count value N has reached the maximum value Nx (step S209). Here, the maximum value Nx corresponds to the number of candidate regions R extracted in step S201. If the count value N is less than the maximum value Nx (if "NO" is the result in step S209), the process returns to step S203, and the count value N is incremented by 1. Steps S203 to S208 are repeated in this manner until the count value N reaches the maximum value Nx, thereby obtaining a non-AI judgment result J1 in which all candidate regions R are determined to be either abnormal regions, uncertain regions, or normal regions. When the count value N reaches the maximum value Nx (if "YES" is the result in step S209), the non-AI detection unit 61 outputs the non-AI judgment result J1 (step S210).

[0047] Once the non-AI detection process (step S101) is completed, step S102 in Figure 5 is executed. In step S102, the image processing unit 51 determines whether an uncertain region Ru exists in the non-AI judgment result J1. If no uncertain region Ru exists (if "NO" is found in step S102), the image processing unit 51 outputs the non-AI judgment result J1 as the inspection result Jr to the detection region synthesis unit 64 (step S103).

[0048] If an uncertain region Ru exists (if "YES" is answered in step S102), the image processing unit 51 executes steps S104 to S105. Figure 8A schematically shows the calculations performed in steps S104 to S106 of the anomaly inspection in Figure 5, and Figure 8B schematically shows the learning content of the trained model used in the AI ​​detection process. As shown in Figure 8B, the trained model 621 learns datasets S1, S2, ... consisting of a training image Il and annotation images Ia indicating the locations of anomalies contained in the training image Il. Therefore, when the image to be inspected Ii is input to the trained model 621, it outputs the locations of anomalies contained in the image to be inspected Ii.

[0049] In other words, in step S104 of the anomaly inspection in Figure 5, the AI ​​detection unit 62 executes the AI ​​detection process. Specifically, the image to be inspected Ii is input to the trained model 621 of the AI ​​detection unit 62, and the trained model 621 outputs the AI ​​detection result Iq (step S104 in Figure 8A). This AI detection result Iq indicates the anomaly location Q contained in the image to be inspected Ii, that is, the anomaly location Q where the chip component is located.

[0050] The AI ​​detection result Iq output by the trained model 621 is input to the detection region limiting unit 63. The detection region limiting unit 63 also obtains uncertain region information Iu from the non-AI detection unit 61. This uncertain region information Iu indicates an uncertain region Ru. The uncertain region information Iu can be any information that indicates an uncertain region Ru, and may be coordinates indicating the location of the uncertain region Ru, or it may be the non-AI judgment result J1 itself. The detection region limiting unit 63 then extracts the AI ​​judgment result J2 indicated by the AI ​​detection result Iq, limited to the uncertain region Ru indicated by the uncertain region information Iu (step S105). In other words, the AI ​​judgment result J2 indicates whether or not the abnormal location Q indicated by the AI ​​detection result Iq is included in the uncertain region Ru. In the example shown in the AI ​​detection result Iql in Figure 8A, the AI ​​judgment result J2 indicates that the abnormal location Q exists in the uncertain region Ru surrounded by a white border, and that the uncertain region Ru is an abnormal region.

[0051] In step S106, the detection region synthesis unit 64 generates an inspection result Jr by combining the non-AI judgment result J1 and the AI ​​judgment result J2. In other words, the judgment result for the region other than the uncertain region Ru from the non-AI judgment result J1 and the judgment result for the uncertain region Ru from the AI ​​judgment result J2 are combined to generate the inspection result Jr. In step S107, the detection region synthesis unit 64 outputs this inspection result Jr.

[0052] In the embodiment described above, an inspection can be performed to detect an anomaly present in the substrate B (object) by performing a non-AI detection process (first anomaly detection process) on the image Ii to be inspected (step S101), and an inspection can be performed to detect an anomaly present in the substrate B by performing an AI detection process (second anomaly detection process) different from the non-AI detection process (step S104) on the image Ii to be inspected (step S104). Therefore, even if the non-AI detection process (step S101) fails to detect an anomaly, the AI ​​detection process (step S104) can successfully detect the anomaly. As a result, it becomes possible to improve the accuracy of detecting anomalies present in the substrate B.

[0053] Furthermore, the non-AI detection process extracts candidate regions R from the image to be inspected Ii by generating a difference image Id and performing image processing that binarizes it using a brightness threshold (step S101). If the area (index value) of the candidate region R is greater than or equal to the threshold area Aa (anomaly detection threshold) (if "YES" is answered in step S204), it is determined that an anomaly exists in the candidate region R (step S205). In contrast, the AI ​​detection process detects anomalies present in candidate regions R (uncertain regions Ru) that have an area less than the threshold area Aa (steps S104, S105). With this configuration, anomalies that could not be detected by the non-AI detection process because they have an index value less than the threshold area Aa can be detected by the AI ​​detection process. As a result, it becomes possible to improve the accuracy of detecting anomalies present in substrate B.

[0054] Furthermore, if the non-AI detection unit 61 extracts a candidate region R (uncertain region Ru) having an area less than the threshold area Aa during the non-AI detection process (if the answer is "YES" in step S102), the AI ​​detection unit 62 and the detection region limiting unit 63 determine whether or not there is an abnormality in the candidate region R with an area less than the threshold area Aa among the candidate regions R extracted by the non-AI detection process (steps S104 to S105). With this configuration, the AI ​​detection process (step S104) is avoided from being executed unnecessarily, and the amount of computation and time required for inspection can be reduced.

[0055] Furthermore, a storage unit 52 is provided to store a reference image Im (reference image) that shows a normal object. The non-AI detection process extracts candidate regions R by performing image processing that compares the inspection target image Ii with the reference image Im (steps S301 to S305). This makes it possible to detect anomalies by the non-AI detection process with a certain degree of accuracy.

[0056] Furthermore, the AI ​​detection process uses a trained model 621, which has been machine-trained on a dataset including a training image Il showing substrate B and annotation images Ia (anomaly training data) indicating anomalies contained in training image Il, to detect anomalies present on substrate B (step S104). In this configuration, anomalies that failed to be detected by the non-AI detection process can be detected by the trained model 621. As a result, it becomes possible to improve the accuracy of detecting anomalies present on substrate B.

[0057] Furthermore, annotation image Ia is data in which the locations where anomalies exist are annotated (Figure 8B). With this configuration, anomaly detection by the trained model 621 can be performed with a certain degree of accuracy.

[0058] Figure 9A is a flowchart showing a second example of non-AI detection processing performed in the anomaly inspection in Figure 5; Figure 9B is a flowchart showing an example of metric value acquisition processing performed in the second example of non-AI detection processing in Figure 9A; Figure 9C is a schematic diagram showing the calculations performed in the metric value acquisition processing in Figure 9B; and Figure 9D is a schematic diagram showing an example of a similarity map obtained in the metric value acquisition processing in Figure 9B.

[0059] In step S401 of the non-AI detection process shown in Figure 9A, an index value acquisition process (Figure 9B) is executed to obtain an index value that represents the similarity between the image and the template image T that indicates the anomaly (chip component). As shown in Figure 9C, this index value acquisition process scans the ROI, exemplified by the white frame, against the image Ii to be inspected, and calculates the similarity between the image within the ROI and the template image T. In this example, two template images T(1) and T(2) are used.

[0060] In step S501 of the index value acquisition process shown in Figure 9B, the non-AI detection unit 61 acquires the inspection target image Ii from the storage unit 52. In step S502, the count value M of the template image T is reset to zero, and in step S503, the count value M is incremented by 1.

[0061] In step S504, the non-AI detection unit 61 starts scanning the ROI for the image Ii to be inspected and calculates the similarity between the image within the ROI and the M-th template image T(M) pixel by pixel. Known methods such as SSD (Sum of Squared Difference) or NCC (Normalized Cross Correlation) can be used to calculate the similarity.

[0062] When the scan is complete (if "YES" is selected in step S506), the non-AI detection unit 61 determines whether the count value M has reached the maximum value Mx (step S507). Here, the maximum value Mx corresponds to the number of template images T. If the count value M is less than the maximum value Mx (if "NO" is selected in step S507), the process returns to step S503 and the count value M is incremented by 1.

[0063] In this way, steps S503 to S506 are repeated until the count value M reaches the maximum value Mx and a "YES" is determined in step S507, thereby calculating the similarity between each template image T and the image Ii to be inspected. When the count value M reaches the maximum value Mx ("YES" in step S507), the non-AI detection unit 61 generates a similarity map MP (Figure 9D) showing the similarity for each pixel (step S508). This similarity map MP may be calculated for each template image T, or it may be configured to output a single similarity map from multiple templates. As an example of a method to output a single similarity map, it can be determined by adopting the largest value among the multiple similarities calculated from multiple template images T as the similarity for each pixel. The following describes the case where a single similarity map is output.

[0064] Once the index value acquisition process (step S401) is completed, the non-AI detection unit 61 executes steps S402 to S411 in Figure 9A. In step S402, the non-AI detection unit 61 extracts candidate regions R from the similarity map MP that consist of pixels with a similarity of threshold similarity Sa or higher. If the corresponding candidate region R is extracted (if "YES" is answered in step S403), the non-AI detection unit 61 determines that the candidate region R extracted in step S402 is an abnormal region (step S404) and proceeds to step S405. On the other hand, if the corresponding candidate region R is not extracted (if "NO" is answered in step S403), the process proceeds directly to step S405. For example, if the threshold similarity Sa is 0.7, the candidate region Rh in the similarity map MP in Figure 9D is determined to be an abnormal region.

[0065] In step S405, the non-AI detection unit 61 extracts candidate regions R from the similarity map MP that consist of pixels with a similarity of less than the threshold similarity Sa and greater than or equal to the threshold similarity Sb. Here, the threshold similarity Sb is smaller than the threshold similarity Sa. If the corresponding candidate region R is extracted (if the answer is "YES" in step S406), the non-AI detection unit 61 determines that the candidate region R extracted in step S405 is an uncertain region (step S407) and proceeds to step S408. On the other hand, if the corresponding candidate region R is not extracted (if the answer is "NO" in step S406), the process proceeds directly to step S408. For example, if the threshold similarity Sa is 0.7, then candidate region Rl in the similarity map MP in Figure 9D is determined to be an uncertain region.

[0066] In step S408, the non-AI detection unit 61 extracts candidate regions R from the similarity map MP that consist of pixels with a similarity of less than the threshold similarity Sb. If the corresponding candidate region R is extracted (if the answer is "YES" in step S409), the non-AI detection unit 61 determines that the candidate region R extracted in step S408 is a normal region (step S410) and proceeds to step S411. On the other hand, if the corresponding candidate region R is not extracted (if the answer is "NO" in step S409), the process proceeds directly to step S411.

[0067] In this way, a non-AI judgment result J1 is obtained, which determines abnormal regions, uncertain regions, or normal regions from the similarity map MP. Then, in step S411, the non-AI detection unit 61 outputs the non-AI judgment result J1.

[0068] In the anomaly inspection using the second example of the non-AI detection process described above (Figure 5), the non-AI detection process (first anomaly detection process) extracts candidate regions R (anomalous regions, uncertain regions, normal regions) from the image to be inspected Ii by performing image processing to calculate the similarity with the template image T. If the similarity (index value) of the candidate regions R is greater than or equal to the threshold similarity Sa (anomaly detection threshold) (if "YES" is found in step S403), it is determined that an anomaly exists in the candidate region R (step S404). The AI ​​detection process (second anomaly detection process) detects anomalies present in candidate regions R (uncertain regions) that have a similarity less than the threshold similarity Sa (step S104). With this configuration, anomalies that could not be detected by the non-AI detection process (step S101) because they have a similarity less than the threshold similarity Sa can be detected by the AI ​​detection process (second anomaly detection process). As a result, it becomes possible to improve the accuracy of detecting anomalies (chip components) present on the substrate B.

[0069] Furthermore, if the non-AI detection unit 61 extracts candidate regions R with a similarity of less than the threshold similarity Sa in the non-AI detection process (if the answer is "YES" in step S102), the AI ​​detection process detects anomalies in the candidate regions R (uncertain regions) with a similarity of less than the threshold similarity Sa extracted by the non-AI detection process (step S104). In this configuration, the AI ​​detection process is avoided from being executed unnecessarily, and the amount of computation and time required for inspection can be reduced.

[0070] Incidentally, the abnormality check performed by the inspection device 5 can be executed at appropriate timings in relation to the component mounting operation by the component mounting machine 2. This point will be explained next.

[0071] Figure 10 is a flowchart showing the mounting start control that performs an abnormality check in relation to the component mounting operation by the component mounting machine. The mounting start control in Figure 10 determines whether or not the component mounting machine 2 will start mounting components onto the substrate B that has been brought in. Here, we will mainly explain the differences from the abnormality check shown in Figure 5, and the explanation of common points will be omitted by indicating corresponding symbols.

[0072] In other words, when board B is loaded into component mounting machine 2, component mounting machine 2 uses board recognition camera Cb to image board B and acquire inspection target image Ii. This inspection target image Ii is transmitted from component mounting machine 2 to inspection device 5, and inspection device 5 performs non-AI detection processing (step S101) on inspection target image Ii.

[0073] If, as a result of the non-AI detection process in step S101, an uncertain region Ru exists (if the answer is "YES" in step S102), the non-AI detection unit 61 performs the AI ​​detection process in step S104 and the region limiting process in step S105 in the same manner as described above, and proceeds to step S106. On the other hand, if no uncertain region Ru exists (if the answer is "NO" in step S102), the non-AI detection unit 61 proceeds directly to step S106.

[0074] In step S106, the non-AI detection unit 61 determines whether the candidate region R, which was determined to be an abnormal region as a result of executing steps S102, S104-S105, exists in the image Ii under inspection. If there is no abnormality (if the result is "NO" in step S106), the non-AI detection unit 61 sends a notification to the component mounting machine 2 authorizing the start of mounting (step S107). As a result, the component mounting machine 2 starts mounting component E onto substrate B. On the other hand, if there is an abnormality (if the result is "YES" in step S106), the non-AI detection unit 61 displays an error notification indicating that there is an abnormality in substrate B on, for example, the display of the UI 53 (step S108).

[0075] Figure 11 is a flowchart showing the in-placement control that performs abnormality checks in relation to the component placement operation by the component placement machine. The in-placement control in Figure 11 is executed after the placement of component E onto board B begins. Here, we will mainly explain the differences from the abnormality check shown in Figure 5, and the explanation of common points will be omitted with corresponding reference numerals.

[0076] In step S121, the arithmetic processing unit 291 determines whether the component E that the mounting head 22 will next mount on the substrate B is the target component. Here, the target component is a predetermined component such as a BGA package component. If it is determined in step S121 that the next component E is the target component (YES in step S121), the arithmetic processing unit 291 acquires an inspection target image Ii by capturing an image of the area where the target component will be mounted using the substrate recognition camera Cb, and transmits the inspection target image Ii to the inspection device 5 (step S122).

[0077] When the inspection device 5 receives the image to be inspected Ii, it executes steps S101, S102, S104-S106 on the image to be inspected Ii in the same manner as described above. In step S106, the non-AI detection unit 61 determines whether the candidate region R, which was determined to be an abnormal region as a result of executing steps S102, S104-S105, exists in the image to be inspected Ii. If there is no abnormality (if "NO" is found in step S106), the non-AI detection unit 61 sends a notification to the component mounting machine 2 allowing the next component E to be mounted on the substrate B (step S123). As a result, the component mounting machine 2 mounts the next component E on the substrate B. On the other hand, if there is an abnormality (if "YES" is found in step S106), the non-AI detection unit 61 displays an error notification indicating that there is an abnormality on the substrate B on, for example, the display of the UI 53 (step S108).

[0078] As mentioned above, the AI ​​detection unit 62 uses the trained model 621 to detect anomalies in the uncertain region Ru of the image Ii under inspection. Next, we will explain the data generation technique for training the trained model 621.

[0079] Figure 12A schematically illustrates a method for generating a dataset for machine learning. The operations shown in Figure 12A are performed, for example, by the image processing unit 51. Specifically, the image processing unit 51 performs image processing to modify a pre-prepared original image Io. Here, the original image Io is acquired by actually imaging the substrate B using a substrate recognition camera Cb or the like, and in this example, it shows a state where chip components are scattered on the substrate B.

[0080] Then, the image processing unit 51 performs image processing on the original image Io to darken the brightness of the chip components, thereby generating a modified image Ic1. The image processing unit 51 then performs image processing on the original image Io to enlarge the chip components, thereby generating a modified image Ic2. Alternatively, the image processing unit 51 then performs image processing on the original image Io to brighten the brightness of areas other than the chip components, i.e., the substrate, thereby generating a modified image Ic3.

[0081] Furthermore, image processing to modify the original image Io is performed using an annotation image Ia (Figure 12B). Here, Figure 12B is a schematic diagram illustrating image modification using an annotation image. As shown in Figure 12B, an annotation image Ia is prepared in advance that shows the location (abnormal location Q) where chip components (i.e., abnormalities) exist in the original image Io. This annotation image Ia is created, for example, by manual work by an operator. In response, the image processing unit 51 recognizes the abnormal location Q indicated by the annotation image Ia from the original image Io.

[0082] Image processing to darken the brightness of chip components is performed by darkening the brightness of the anomaly area Q indicated in annotation image Ia. For example, methods such as uniformly subtracting or dividing the brightness of each pixel within the area of ​​anomaly area Q can be considered. Image processing to enlarge chip components is performed by increasing the size of the anomaly area Q indicated in annotation image Ia. Image processing to brighten the brightness of the substrate is performed by brightening the brightness of the area outside of the anomaly area Q indicated in annotation image Ia. This allows for the generation of multiple datasets of annotation image Ia and modified image Ic.

[0083] Furthermore, for modified images Ic1 and Ic3, which are generated by changing the brightness, the annotation image Ia of the original image Io can be used as is. On the other hand, the annotation image Ia for modified image Ic2, which is generated by changing the size of the chip components, can be generated by increasing the abnormal area Q of the annotation image Ia for the original image Io.

[0084] Furthermore, the anomalies to be detected are not limited to the scattering of chip components on substrate B, but may also include, for example, the adhesion of solder bridges to the leads of components. A dataset for machine learning can be generated for these anomalies in the same manner.

[0085] Figures 13A and 13B schematically illustrate a method for generating a dataset for learning solder bridge anomalies. In this example, when the original image Io is input to the image processing unit 51, the image processing unit 51 identifies the lead regions where component leads are provided from the original image Io. This may be identified by pattern matching or by information indicating the configuration of the component.

[0086] Then, for two adjacent leads, point ds is set on one lead and point de is set on the other lead. This setting may be performed by the image processing unit 51 in response to an operation on the operator UI 53, or it may be performed automatically by the image processing unit 51 based on a predetermined program. As a result, in the example in Figure 13A, points ds1 and de1 are set, and in the example in Figure 13B, points ds2 and de2 are set.

[0087] The image processing unit 51 then modifies the original image Io by adding an image of a solder bridge connecting points ds and de, thereby generating a modified image Ic. The image processing unit 51 also generates an annotation image Ia that indicates the location of the solder bridge (anomalous area Q). In this way, a dataset of the modified image Ic and the annotation image Ia is generated for learning solder bridge anomalies.

[0088] In these examples, an original image Io (training image) showing the target object, such as substrate B or components, is obtained. Then, by modifying the original image Io, a modified image Ic different from the original image Io is obtained. In addition, an annotation image Ia (anomaly training data) showing the modified image Ic and the anomaly Q contained in the modified image Ic is obtained. Then, machine learning is performed on the trained model 621 using the dataset containing the modified image Ic and annotation image Ia. Therefore, many datasets can be prepared and used to train the trained model 621. As a result, the accuracy of the trained model 621 can be improved.

[0089] As described above, in this embodiment, substrate B corresponds to an example of the "object" of the present invention, inspection target image Ii corresponds to an example of the "inspection target image" of the present invention, substrate recognition camera Cb corresponds to an example of the "camera" of the present invention, non-AI detection processing (step S101) corresponds to an example of the "first anomaly detection processing" of the present invention, non-AI detection unit 61 corresponds to an example of the "first anomaly detection unit" of the present invention, AI detection processing (step S104) corresponds to an example of the "second anomaly detection processing" of the present invention, and AI detection unit 62 corresponds to an example of the "second anomaly detection unit" of the present invention. The inspection device 5 corresponds to an example of the "inspection device" of the present invention, the reference image Im corresponds to an example of the "reference image" of the present invention, the storage unit 52 corresponds to an example of the "storage unit" of the present invention, the trained model 621 corresponds to an example of the "trained model" of the present invention, the component mounting machine 2 corresponds to an example of the "component mounting machine" of the present invention, the production system 1 corresponds to an example of the "production system" of the present invention, the inspection program Pi corresponds to an example of the "inspection program" of the present invention, and the external server computer or storage unit 52 corresponds to an example of the "recording medium" of the present invention.

[0090] It should be noted that the present invention is not limited to the embodiments described above, and various modifications can be made to the above without departing from its spirit. For example, in the above anomaly inspection, the trained model 621 outputs an AI detection result Iq for the image to be inspected Ii. Therefore, the parameters of the trained model 621 may be updated by performing additional training using a dataset that includes the image to be inspected Ii and data in which the AI ​​detection result Iq has been manually corrected. Alternatively, the parameters of the trained model 621 may be updated by training the trained model 621 with a dataset that includes the image to be inspected Ii and an annotation image Ia that indicates the anomaly location Q contained in the image to be inspected Ii. In this case, the target of training may be all anomalies included in the AI ​​detection result Iq, or it may be only anomalies included in the uncertain region Ru. With such a configuration, the accuracy of the trained model 621 can be improved through the execution of anomaly inspection.

[0091] Furthermore, the inspection device may be configured such that the second anomaly detection unit updates the trained model 621 by additionally training it with the image to be inspected and a dataset containing data on anomalies contained in the image to be inspected.

[0092] Furthermore, in the above example, the detection area limiting unit 63 extracts the AI ​​judgment result J2 indicated by the AI ​​detection result Iq, but only within the uncertain area Ru indicated by the uncertain area information Iu. However, the target area (limiting area) for extraction from the AI ​​detection result Iq does not have to be the uncertain area Ru itself. Any area that includes the uncertain area Ru and is narrower than the inspection target image Ii may be set as the limiting area, and the judgment result included in the limiting area of ​​the AI ​​detection result Iq may be extracted.

[0093] In this case, a figure of a predetermined shape centered on the geometric centroid of the uncertain region Ru may be set as the limiting region. Examples of such figures include polygons such as squares, circles, or ellipses. Alternatively, a rectangle in which each of the four sides is circumscribed with the uncertain region Ru may be set as the limiting region.

[0094] Furthermore, during component mounting in the component mounting machine 2, the orientation of component E is determined by the component recognition image Ir captured by the component recognition camera Ce. At this time, component E with a poor orientation is highly likely to fall from the nozzle. Therefore, when a component E with a poor orientation is detected, a threshold area Aa or threshold area Ab may be set based on the configuration of component E, and the determination in step S204 or S206 may be performed. Alternatively, a template image T may be set based on the configuration of component E, and the similarity may be determined in Figure 9B.

[0095] In the above example, the trained model 621 performs AI judgment processing on the entire image Ii to be inspected and outputs an AI detection result Iq, and the detection region limiting unit 63 limits it to the uncertain region Ru and extracts the judgment result from the AI ​​detection result Iq. However, the trained model 621 may be configured to limit the AI ​​judgment processing to the uncertain region Ru of the image Ii to be inspected and obtain the AI ​​judgment result J2. In this case, by setting the input image size during AI training to the size of the uncertain region which has been set in advance, it is possible to speed up the AI ​​processing and improve the accuracy of detecting anomalies.

[0096] Furthermore, it is not necessary to provide the functions of the inspection device 5 separately from the component mounting machine 2. Therefore, the non-AI detection unit 61, AI detection unit 62, detection area limiting unit 63, and detection area synthesis unit 64 may be configured in the arithmetic processing unit 291 of the component mounting machine 2, and the inspection target image Ii and reference image Im may be stored in the storage unit 292. In this case, the component mounting machine 2 performs the above-mentioned abnormality inspection. [Explanation of Symbols]

[0097] 1…Production system 2…Component mounting machine 5…Inspection equipment 52...Storage section 61...Non-AI detection unit 62...AI detection unit 621... Pre-trained model B... Circuit board Cb... Circuit board recognition camera II...Image to be examined Im... (Reference image) Pi... Test Program

Claims

1. A camera that images an object and acquires an image of the object to be inspected, A first anomaly detection unit detects an anomaly present in the object by performing a first anomaly detection process on the image to be inspected, A second anomaly detection unit detects an anomaly present in the object by performing a second anomaly detection process different from the first anomaly detection process on the image to be inspected. An inspection device equipped with the following features.

2. The first anomaly detection process extracts candidate regions from the image to be inspected by performing predetermined image processing, and if the index value of the degree of anomaly in the candidate region is equal to or greater than the anomaly detection threshold, it is determined that an anomaly exists in the candidate region. The inspection apparatus according to claim 1, wherein the second anomaly detection process detects an anomaly that exists in a predetermined limited area narrower than the image to be inspected, including the candidate area having the index value less than the anomaly detection threshold.

3. The inspection apparatus according to claim 2, wherein when the first anomaly detection unit extracts candidate regions having an index value less than the anomaly detection threshold in the first anomaly detection process, the second anomaly detection unit detects an anomaly in a predetermined limited region that is narrower than the inspection target image, including the candidate regions with an index value less than the anomaly detection threshold among the candidate regions extracted by the first anomaly detection process.

4. The inspection apparatus according to claim 3, wherein the index value is area.

5. The inspection apparatus according to claim 3, wherein the index value is the similarity to a predetermined pattern.

6. The inspection apparatus according to claim 2, wherein the candidate region below the abnormality detection threshold is set as the limited region.

7. It further includes a storage unit that stores a reference image showing a normal object, The inspection apparatus according to claim 2, wherein the first anomaly detection process extracts the candidate region by performing the image processing that compares the image to be inspected with the reference image.

8. The inspection apparatus according to claim 1, wherein the second anomaly detection process detects an anomaly present in the object using a trained model that has been trained on a dataset including a training image showing the object and anomaly teaching data indicating anomalies contained in the training image.

9. The inspection apparatus according to claim 8, wherein the abnormality teaching data is data in which the location of the abnormality is annotated.

10. The inspection apparatus according to claim 8, wherein the second anomaly detection unit performs additional learning using the image to be inspected and a dataset containing data relating to anomalies contained in the image to be inspected, and updates the trained model.

11. An inspection apparatus according to any one of claims 1 to 10, A mounting head for mounting components onto the aforementioned object and A component mounting machine equipped with the following features.

12. An inspection apparatus according to any one of claims 1 to 10, A component mounting machine for mounting components onto the aforementioned object and A production system equipped with the following features.

13. A step of capturing an image of the object using a camera to obtain an image of the object to be inspected, The process involves performing a first anomaly detection process on the image to be inspected to detect an anomaly present in the object, A step of detecting an abnormality present in the object by performing a second abnormality detection process, which is different from the first abnormality detection process, on the image to be inspected. An inspection method that includes the following features.

14. A method for generating a trained model used in an inspection method that detects anomalies present in an object using a trained model, A step of acquiring an original image showing the aforementioned object, A step of obtaining a modified image different from the original image by modifying the original image, A step of having the trained model perform machine learning on a dataset that includes the modified image and anomaly training data indicating anomalies contained in the modified image. A method for generating a pre-trained model that includes [a specific feature / ability].

15. An inspection program that causes a computer to execute the inspection method described in claim 13.

16. A recording medium for recording the inspection program described in claim 15 in a manner readable by a computer.

Images