Solder fillet inspection device and solder fillet inspection method

Abstract

To reduce the effort and burden required to obtain identification means as an AI model, and also makes it possible to commonly use the identification means even when the land sizes are different.SOLUTION: AI models 101, 102 are generated by training a specific neural network using only image data related to good solder fillets as training data, and inspection image data based on image data acquired by a camera 32d is input to the AI models to obtain reconstructed image data, and the quality of the solder fillets is determined by comparing the inspection image data and the reconstructed image data. The training data is formed by providing one solder fillet image in an image frame that is larger than the size of the one solder fillet image. The inspection image data is formed by providing one solder fillet image extracted from image data acquired by the camera 32d in an image frame that is the same size as the image frame of the training data.SELECTED DRAWING: Figure 5

Description

Technical Field

[0001] The present invention relates to an inspection apparatus and an inspection method for inspecting a solder fillet for soldering an electronic component.

Background Art

[0002] Generally, in a substrate manufacturing line for mounting electronic components on a printed circuit board, first, cream solder is printed on the lands of the printed circuit board (solder printing process). Next, the electronic components are temporarily fixed on the printed circuit board based on the viscosity of the cream solder (mounting process). Then, the printed circuit board is led into a reflow furnace, and soldering is performed by heating and melting the cream solder (reflow process). In such a substrate manufacturing line, an inspection apparatus for inspecting the printed circuit board may be provided.

[0003] Recently, as an inspection apparatus for inspecting the cream solder after the reflow process, that is, the solder fillet for soldering an electronic component, an apparatus using an AI model has been proposed (see, for example, Patent Document 1). This inspection apparatus measures a predetermined index based on an inspection image by executing a predetermined inspection program, and inspects the state of the inspection target using the measured value. The inspection program executed generates an AI model for each type of electronic component (component type). Here, in many cases, since the size of the land on which the electronic component is mounted varies depending on the type of the electronic component, it can be said that this inspection program generates different AI models for different land sizes.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, since the sizes of the lands vary, it is necessary to perform complicated operations to generate and prepare different AI models for each land size, which may require a great deal of labor and effort.

[0006] The present invention has been made in view of the above circumstances, and its object is to reduce the labor and burden in obtaining the discrimination means as an AI model, and to provide a solder fillet inspection device or the like that can commonly use the discrimination means even when the land sizes are different.

Means for Solving the Problems

[0007] Hereinafter, each means suitable for solving the above object will be described separately. In addition, the effects specific to the corresponding means will be appended as necessary.

[0008] Means 1. A solder fillet inspection device for inspecting a solder fillet for soldering an electronic component on a printed circuit board, image data acquisition means capable of acquiring image data of a predetermined inspection area on the printed circuit board including the solder fillet, discrimination means generated by learning only the image data related to a good solder fillet as learning data for a neural network having an encoding unit that extracts feature amounts from the input image data and a decoding unit that reconstructs the image data from the feature amounts, inspection image data acquisition means for acquiring inspection image data including an image of the solder fillet to be inspected based on the image data acquired by the image data acquisition means, reconstructed image data acquisition means capable of acquiring, as reconstructed image data, the image data reconstructed by inputting the inspection image data to the discrimination means, comparison means capable of comparing the inspection image data and the reconstructed image data, configured to be able to determine the quality of the solder fillet based on the comparison result by the comparison means, The learning data is formed by providing a solder fillet image of 1, which shows a solder fillet corresponding to a land of 1, in an image frame having a size larger than the size of the solder fillet image of 1. The inspection image data acquisition means acquires the inspection image data of the same size as the learning data, which is formed by providing the solder fillet image of 1 extracted from the image data acquired by the image data acquisition means in an image frame having the same size as the image frame of the learning data. A solder fillet inspection apparatus characterized by the above.

[0009] In addition, the solder fillet image of 1 may be an image (image of means 2 described later) showing the entire connected component (block part) of the solder fillet at least partially located on a land of 1, or an image showing only the part of the connected component of the solder fillet that exists on a land of 1. Further, the solder fillet image of 1 may include lands, electrodes, etc. located around the connected component of the solder fillet in addition to the connected component of the solder fillet.

[0010] In addition, the solder fillet image of 1 constituting the learning data may be extracted from image data (actual image data) obtained by imaging a printed circuit board provided with a good solder fillet (that is, an image of an actual solder fillet), or may be an image of a virtual good solder fillet. Examples of the actual image data include image data accumulated in previous inspections, and image data of a good printed circuit board visually selected by an operator after the reflow process.

[0011] Furthermore, the above "neural network" includes, for example, a convolutional neural network having a plurality of convolutional layers. The above "learning" includes, for example, deep learning. The above "discrimination means (generation model)" includes, for example, an autoencoder (self-encoder) and a convolutional autoencoder (convolutional self-encoder).

[0012] In addition, the "identification means" is generated by learning only the image data related to the solder fillet of good products (learning data in which one solder fillet image related to the solder fillet of good products is provided in the image frame). Therefore, when the inspection image data related to the solder fillet of defective products is input to the identification means, the reconstructed image data generated is almost the same as the inspection image data in which the defective part has been corrected (for example, the shape and area are made correct). That is, when there is a defective part in the solder fillet, virtual image data related to the solder fillet assuming that there is no defective part is generated as the reconstructed image data related to the solder fillet.

[0013] According to the above means 1, the inspection image data is formed by providing one solder fillet image extracted from the image data acquired by the image data acquisition means in the image frame. Therefore, the size (width and height) of the inspection image data does not vary finely depending on the size of the land and becomes constant. As a result, it is not necessary to prepare a large number of different identification means for each land size, and the labor and effort required to obtain the identification means can be reduced. In addition, the identification means can be commonly used even when the land sizes are different.

[0014] Furthermore, according to the above means 1, the image frame of the learning data and the image frame of the inspection image data are the same size, and the sizes of the learning data and the inspection image data are the same. Therefore, when the inspection image data is input to the identification means, appropriate reconstructed image data corresponding to the inspection image data can be more reliably output, and as a result, the quality determination of the solder fillet can be performed more accurately. Thereby, good inspection accuracy can be obtained more reliably.

[0015] In addition, according to the above means 1, inspection image data and reconstructed image data that is input into the identification means and reconstructed from the inspection image data are compared, and based on the comparison result, the quality of the solder fillet is determined. Therefore, both pieces of image data to be compared are related to the same solder fillet. Accordingly, unlike the method of determining quality by comparison with a separately prepared standard, there is no need to set relatively loose inspection conditions to prevent false detection, and stricter inspection conditions can be set. Furthermore, in both pieces of image data to be compared, the imaging conditions of the printed circuit board to be inspected (for example, the placement position, placement angle, deflection, etc. of the printed circuit board) and the imaging conditions on the inspection apparatus side (for example, the illumination state, the angle of view of the camera, etc.) can be made to match. Together, these enable the quality determination of the solder fillet to be performed more accurately.

[0016] Means 2. A solder fillet image extraction means for extracting the solder fillet image of 1 that constitutes the inspection image data from the image data acquired by the image data acquisition means is provided. The solder fillet image extraction means is capable of identifying the region occupied by the solder fillet in the image data acquired by the image data acquisition means and extracting the image of the connected component in the identified region as the solder fillet image of 1 that constitutes the inspection image data. The solder fillet inspection apparatus according to means 1, characterized by this.

[0017] According to the above-described means 2, the area occupied by the solder fillet in the image data acquired by the image data acquisition means is specified by the solder fillet image extraction means, and the image of the connected component in the specified area is extracted as one solder fillet image constituting the inspection image data. Therefore, one solder fillet image includes not only the image of the portion located on the land in the solder fillet but also the image of the portion protruding from the land in the solder fillet. That is, as shown in FIG. 27, even if the solder fillet 5 protrudes from the land 3, as shown in FIG. 28, one solder fillet image Ih includes this protruding portion. Thereby, it becomes possible to appropriately perform the pass / fail determination regarding the solder fillet partially protruding from the land, and the inspection accuracy can be further improved.

[0018] Means 3. Second discrimination means is provided, which is generated by training only the image data related to the good solder fillets as the second training data for a neural network having an encoding unit that extracts feature amounts from the input image data and a decoding unit that reconstructs the image data from the feature amounts. The second training data is obtained by providing the one solder fillet image in a second image frame having a size larger than the size of the one solder fillet image and smaller than the size of the image frame of the training data. When the size of the one solder fillet image extracted from the image data acquired by the image data acquisition means is smaller than the size of the second image frame, the inspection image data acquisition means acquires the inspection image data having the same size as the second training data, which is obtained by providing the one solder fillet image in the second image frame. The reconstructed image data acquisition means acquires the reconstructed image data reconstructed by inputting the inspection image data to the second discrimination means. The comparison means is configured to compare the inspection image data and the reconstructed image data. The solder fillet inspection apparatus according to means 1, characterized in that.

[0019] According to the above means 3, when the size of the solder fillet image of 1 is relatively small, inspection image data acquisition means provides the solder fillet image of 1 in a relatively small second image frame, and relatively small-sized inspection image data is acquired. Then, when this relatively small inspection image data is input to the second identification means by the reconstructed image data acquisition means, reconstructed image data is output, and the relatively small inspection image data and the reconstructed image data are compared by the comparison means. Therefore, compared with the case where the image frame of the inspection image data is always set to a constant size, the processing for acquiring the reconstructed image data and the comparison processing by the comparison means can be speeded up, and thus the inspection speed can be further improved.

[0020] Means 4. The learning data and the inspection image data are set such that the center or centroid of the solder fillet image of 1 coincides with the center of the image frame, and the part on the electronic component side in the solder fillet image of 1 faces a predetermined direction. The solder fillet inspection apparatus according to means 1, characterized in that.

[0021] In addition, the technical matters related to the above means 4 may be applied to the above means 3. That is, the second learning data may be set such that the center or centroid of the solder fillet image of 1 coincides with the center of the second image frame, and the part on the electronic component side in the solder fillet image of 1 faces a predetermined direction. Of course, the inspection image data related to the above means 3 may be set in the same manner.

[0022] According to the above means 4, the orientation and position of the solder fillet in the learning data and the inspection image data can be generally aligned. Therefore, even if the learning data used for generating the identification means is relatively small, the quality of the solder fillet can be accurately determined. That is, good inspection accuracy can be obtained while more effectively reducing the labor and effort required to obtain the identification means.

[0023] Means 5. The comparison means is configured to be able to compare the inspection image data and the reconstructed image data by using only the solder fillet image of 1 in the inspection image data as a comparison target, which is a feature of the solder fillet inspection apparatus according to Means 1.

[0024] According to the above Means 5, the comparison means compares the inspection image data and the reconstructed image data by using only the solder fillet image of 1 in the inspection image data as a comparison target. That is, when comparing the two image data, the comparison means does not use the parts other than the solder fillet image of 1 in the inspection image data as comparison targets. Therefore, compared with the case of comparing the entire two image data, the processing load related to the comparison of the two image data can be reduced, and the inspection can be speeded up and made more efficient. In addition, it is possible to more surely prevent the parts other than the solder fillet image of 1 in the inspection image data, that is, the parts unrelated to the solder fillet, from affecting the pass / fail determination, and thus it is possible to further improve the inspection accuracy.

[0025] Means 6. A solder fillet inspection method for inspecting a solder fillet for soldering an electronic component on a printed circuit board, comprising: an image data acquisition step capable of acquiring image data of a predetermined inspection area on the printed circuit board including the solder fillet; an inspection image data acquisition step of acquiring inspection image data including an image of the solder fillet to be inspected based on the image data acquired in the image data acquisition step; a reconstruction image data acquisition step capable of acquiring, as reconstruction image data, the image data reconstructed by inputting the inspection image data acquired in the inspection image data acquisition step to an identification means generated by learning only the image data related to a good solder fillet as learning data for a neural network having an encoding unit for extracting feature amounts from the input image data and a decoding unit for reconstructing the image data from the feature amounts; and a comparison step of comparing the inspection image data and the reconstruction image data. Based on the comparison result in the comparison step, determine the quality of the solder fillet. The learning data is formed by providing a solder fillet image of 1 corresponding to a land of 1 in an image frame having a size larger than the size of the solder fillet image of 1. In the inspection image data acquisition step, inspection image data of the same size as the learning data is acquired, wherein the solder fillet image of 1 extracted from the image data acquired in the image data acquisition step is provided in an image frame of the same size as the image frame of the learning data. A solder fillet inspection method characterized by the above.

[0026] According to the above means 6, the same operational effects as those of the above means 1 are achieved.

[0027] In addition, the technical matters related to the above respective means may be combined as appropriate. Therefore, for example, the technical matters related to the above means 3 etc. may be combined with respect to the technical matters related to the above means 2. Also, for example, at least one of the technical matters related to the above means 2 to 5 may be applied to the above means 6.

Brief Description of Drawings

[0028]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

Figure 25

Figure 26

Figure 27

Figure 28

Mode for Carrying Out the Invention

[0029] Hereinafter, an embodiment will be described with reference to the drawings. First, the configuration of the printed circuit board will be described. FIG. 1 is a partially enlarged plan view of a part of the printed circuit board.

[0030] As shown in FIG. 1, the printed circuit board 1 is formed by forming a wiring pattern (not shown) made of copper foil and a plurality of lands 3 on the surface of a flat base substrate 2 made of glass epoxy resin or the like. A resist film 4 is coated on the surface of the base substrate 2 excluding the lands 3.

[0031] Furthermore, an electronic component 6 such as a chip is mounted on the land 3 via a solder fillet 5. In FIG. 1 and the like, for the sake of convenience, a scatter pattern is attached to the part showing the solder fillet 5. The solder fillet 5 is formed by heating cream solder made by kneading solder grains with flux in a reflow process described later. The solder fillet 5 is for soldering the electronic component 6 to the printed circuit board 1 and joins the electrode 6a and the land 3. In the present embodiment, as the solder fillet 5, relatively large solder fillets 5a and 5b provided on relatively large lands 3 and corresponding to relatively large electronic components 6, and relatively small solder fillets 5c and 5d provided on relatively small lands 3 and corresponding to relatively small electronic components 6 exist.

[0032] Next, the manufacturing line (manufacturing process) for manufacturing the printed circuit board 1 will be described with reference to FIG. 2. FIG. 2 is a block diagram showing the configuration of the manufacturing line 10 for the printed circuit board 1. As shown in FIG. 2, in the manufacturing line 10, in order from its upstream side (upper side in FIG. 2), a solder printer 12, a solder printing state inspection device 13, a component mounter 14, a reflow device 15, and a solder fillet inspection device 16 are installed.

[0033] The solder printer 12 performs a solder printing process for printing cream solder on each land 3 of the printed circuit board 1. In the solder printing process, for example, cream solder is printed by screen printing. In screen printing, first, with the lower surface of the screen mask in contact with the printed circuit board 1, cream solder is supplied to the upper surface of the screen mask. A plurality of openings corresponding to each land 3 of the printed circuit board 1 are formed in the screen mask. Next, by moving while bringing a predetermined squeegee into contact with the upper surface of the screen mask, the openings are filled with cream solder. Thereafter, by separating the printed circuit board 1 from the lower surface of the screen mask, cream solder is printed on each land 3 of the printed circuit board 1.

[0034] The solder printing state inspection device 13 inspects the shape of the cream solder printed on the land 3 and the presence or absence of foreign matter adhering to the cream solder.

[0035] The component mounter 14 mounts the electronic component 6 on the land 3 on which the cream solder is printed. By the component mounter 14, the electrodes 6a of the electronic component 6 are temporarily fixed to the respective predetermined cream solders.

[0036] The reflow device 15 performs a reflow process of soldering (soldering) the land 3 and the electrode 6a of the electronic component 6 by heating and melting the cream solder. By the reflow process, a solder fillet 5 formed by heating the cream solder is formed, and as a result, the land 3 and the electronic component 6 (electrode 6a) are fixed.

[0037] The solder fillet inspection device 16 inspects whether the soldering (soldering attachment) is properly performed in the reflow process by inspecting the shape, area, amount, etc. of the solder fillet 5. The solder fillet inspection device 16 will be described in detail later.

[0038] Although not shown in the figure, the manufacturing line 10 is provided with a conveyor or the like for transferring the printed circuit board 1 between the above-mentioned devices such as between the solder printer 12 and the solder printing state inspection device 13. A branching device is provided between the solder printing state inspection device 13 and the component mounter 14 and on the downstream side of the solder fillet inspection device 16. The printed circuit board 1 determined to be a good product by the solder printing state inspection device 13 or the solder fillet inspection device 16 is guided directly to the downstream side, while the printed circuit board 1 determined to be a defective product is discharged to the defective product storage section by the branching device.

[0039] Next, the configuration of the solder fillet inspection device 16 will be described in detail with reference to FIGS. 3 and 4. FIG. 3 is a schematic configuration diagram schematically showing the solder fillet inspection device 16. FIG. 4 is a block diagram showing the functional configuration of the solder fillet inspection device 16.

[0040] The solder fillet inspection device 16 includes a transport mechanism 31 for transporting and positioning the printed circuit board 1, an inspection unit 32 for obtaining image data of the printed circuit board 1, and a control device 33 (see FIG. 4) that executes various controls, image processing, and arithmetic processing in the solder fillet inspection device 16, including driving control of the transport mechanism 31 and the inspection unit 32.

[0041] The transport mechanism 31 includes a pair of transport rails 31a arranged along the loading / unloading direction of the printed circuit board 1, and an endless conveyor belt 31b rotatably disposed with respect to each transport rail 31a. Although not shown in the figure, the transport mechanism 31 is provided with driving means such as a motor for driving the conveyor belt 31b and a chuck mechanism for positioning the printed circuit board 1 at a predetermined position. The transport mechanism 31 is driven and controlled by a control device 33 (a transport mechanism control unit 79 described later).

[0042] Under the above configuration, the printed circuit board 1 carried into the solder fillet inspection device 16 has both side edges in the width direction perpendicular to the carry-in / carry-out direction inserted into the conveying rails 31a, respectively, and is placed on the conveyor belt 31b. Subsequently, the conveyor belt 31b starts operating, and the printed circuit board 1 is conveyed to a predetermined inspection position. When the printed circuit board 1 reaches the inspection position, the conveyor belt 31b stops and the chuck mechanism operates. By the operation of this chuck mechanism, the conveyor belt 31b is pushed up, and both side edges of the printed circuit board 1 are clamped by the upper side portions of the conveyor belt 31b and the conveying rails 31a. As a result, the printed circuit board 1 is positioned and fixed at the inspection position. When the inspection is completed, the fixing by the chuck mechanism is released, and the conveyor belt 31b starts operating. Thereby, the printed circuit board 1 is carried out from the solder fillet inspection device 16. Of course, the configuration of the conveying mechanism 31 is not limited to the above form, and other configurations may be adopted.

[0043] The inspection unit 32 is disposed above the conveying rail 31a (the conveying path of the printed circuit board 1). The inspection unit 32 includes a first lighting device 32a, a second lighting device 32b, a third lighting device 32c, and a camera 32d. In the present embodiment, the camera 32d constitutes the "image data acquisition means".

[0044] Further, the inspection unit 32 also includes an X-axis moving mechanism 32e (see FIG. 4) that enables movement in the X-axis direction (the left-right direction in FIG. 3), and a Y-axis moving mechanism 32f (see FIG. 4) that enables movement in the Y-axis direction (the front-rear direction in FIG. 3). These moving mechanisms 32e and 32f are driven and controlled by a control device 33 (a moving mechanism control unit 76 described later).

[0045] When performing three-dimensional measurement of the printed circuit board 1, the first lighting device 32a and the second lighting device 32b irradiate a predetermined inspection area on the printed circuit board 1 with predetermined light for three-dimensional measurement (pattern light having a striped light intensity distribution) from obliquely above, respectively.

[0046] Specifically, the first lighting device 32a includes a first light source 32a1 that emits predetermined light, and a first liquid crystal shutter 32a2 that forms a first grating for converting the light from the first light source 32a1 into first pattern light having a striped light intensity distribution. It is driven and controlled by a control device 33 (a lighting control unit 72 described later).

[0047] The second lighting device 32b includes a second light source 32b1 that emits predetermined light, and a second liquid crystal shutter 32b2 that forms a second grating for converting the light from the second light source 32b1 into second pattern light having a striped light intensity distribution. It is driven and controlled by a control device 33 (a lighting control unit 72 described later).

[0048] Under the above configuration, the light emitted from each of the light sources 32a1 and 32b1 is respectively guided to a condenser lens (not shown), made into parallel light there, and then guided to a projection lens (not shown) through the liquid crystal shutters 32a2 and 32b2, and projected onto the printed circuit board 1 as pattern light. Further, in the present embodiment, the switching control of the liquid crystal shutters 32a2 and 32b2 is performed so that the phases of the respective pattern lights are shifted by a quarter pitch each.

[0049] In addition, by using the liquid crystal shutters 32a2 and 32b2 as gratings, it is possible to irradiate pattern light close to an ideal sine wave. Thereby, the measurement resolution of three-dimensional measurement is improved. Also, the phase shift control of the pattern light can be performed electrically, and the device can be made compact.

[0050] When performing two-dimensional measurement of the printed circuit board 1, the third lighting device 32c irradiates a predetermined inspection area on the printed circuit board 1 with predetermined light (for example, uniform light) for two-dimensional measurement. The third lighting device 32c includes a ring light capable of irradiating blue light, a ring light capable of irradiating green light, and a ring light capable of irradiating red light. Since the third lighting device 32c has the same configuration as the known technology, a detailed description thereof will be omitted.

[0051] The camera 32d images a predetermined inspection area of the printed circuit board 1 from directly above. The camera 32d has an imaging element such as a CCD (Charge Coupled Device) type image sensor or a CMOS (Complementary Metal Oxide Semiconductor) type image sensor, and an optical system (such as a lens unit and an aperture) that forms an image of the printed circuit board 1 on the imaging element, and is arranged such that its optical axis is along the vertical direction (Z-axis direction). Of course, the imaging element is not limited to these, and other imaging elements may be adopted.

[0052] The camera 32d is driven and controlled by a control device 33 (a camera control unit 73 described later). More specifically, the control device 33 executes imaging processing by the camera 32d while synchronizing with the irradiation processing by each of the lighting devices 32a, 32a, 32c. As a result, among the light irradiated from any of the lighting devices 32a, 32b, 32c, the light reflected by the printed circuit board 1 is imaged by the camera 32d. As a result, image data of the inspection area of the printed circuit board 1 including the solder fillet 5 is acquired. Note that the "inspection area" of the printed circuit board 1 is one area among a plurality of areas preset on the printed circuit board 1 with the size of the imaging field (imaging range) of the camera 32d as one unit.

[0053] Also, the camera 32d in the present embodiment is configured as a color camera. As a result, it is possible to simultaneously image the light of each color irradiated from each color ring light of the third lighting device 32c and reflected by the printed circuit board 1.

[0054] The image data imaged and generated by the camera 32d is converted into a digital signal inside the camera 32d and then transferred to the control device 33 (an image acquisition unit 74 described later) in the form of a digital signal. Then, the control device 33 stores the transferred image data and performs various image processing and arithmetic processing based on the image data.

[0055] The control device 33 consists of a computer including a CPU (Central Processing Unit) that executes predetermined arithmetic processing, a ROM (Read Only Memory) that stores various programs and fixed-value data, etc., a RAM (Random Access Memory) that temporarily stores various data when executing various arithmetic processes, and peripheral circuits thereof.

[0056] And, when the CPU of the control device 33 operates according to various programs, it functions as various functional units such as a main control unit 71, a lighting control unit 72, a camera control unit 73, an image acquisition unit 74, a data processing unit 75, a movement mechanism control unit 76, a learning unit 77, an inspection unit 78, and a transport mechanism control unit 79.

[0057] However, the above various functional units are realized by the cooperation of various hardware such as the above CPU, ROM, and RAM, and there is no need to clearly distinguish between functions realized in hardware or software. Some or all of these functions may be realized by a hardware circuit such as an IC.

[0058] Furthermore, the control device 33 is provided with an input unit 55 composed of a keyboard, a mouse, a touch panel, etc., a display unit 56 having a display screen composed of a liquid crystal display, etc., a storage unit 57 capable of storing various data, programs, calculation results, inspection results, etc., a communication unit 58 capable of transmitting and receiving various data to and from the outside, and the like.

[0059] Here, the above various functional units constituting the control device 33 will be described in detail.

[0060] The main control unit 71 is a functional unit that controls the entire solder fillet inspection device 16, and is configured to be able to transmit and receive various signals to and from other functional units such as the lighting control unit 72 and the camera control unit 73.

[0061] The lighting control unit 72 is a functional unit that drives and controls the lighting devices 32a, 32b, and 32c, and performs switching control of irradiation light and the like based on a command signal from the main control unit 71.

[0062] The camera control unit 73 is a functional unit that drives and controls the camera 32d, and controls imaging timing and the like based on a command signal from the main control unit 71.

[0063] The image acquisition unit 74 is a functional unit for capturing the image data captured and acquired by the camera 32d.

[0064] The data processing unit 75 is a functional unit that performs predetermined image processing on the image data captured by the image acquisition unit 74, or performs two-dimensional measurement processing, three-dimensional measurement processing, etc. using the image data.

[0065] The movement mechanism control unit 76 is a functional unit that drives and controls the X-axis movement mechanism 32e and the Y-axis movement mechanism 32f, and controls the position of the inspection unit 32 based on a command signal from the main control unit 71. By driving and controlling the X-axis movement mechanism 32e and the Y-axis movement mechanism 32f, the movement mechanism control unit 76 can move the inspection unit 32 to a position above an arbitrary inspection area of the printed circuit board 1 fixed at the inspection position. Then, while the inspection unit 32 is sequentially moved to a plurality of inspection areas set on the printed circuit board 1, the inspection related to the inspection area is executed, thereby performing the inspection of the entire printed circuit board 1.

[0066] The learning unit 77 is a functional unit that performs learning of the deep neural network 90 (hereinafter simply referred to as "neural network 90"; see FIG. 5) using learning data, and constructs the first AI (Artificial Intelligence) model 101 as "discrimination means" and the second AI model 102 as "second discrimination means".

[0067] In addition, as will be described later, each of the AI models 101 and 102 in the present embodiment is a generation model constructed by causing the neural network 90 to perform deep learning (deep learning) using only the image data related to the good solder fillet 5 as learning data, and has a structure of a so-called autoencoder (autoencoder).

[0068] Here, the structure of the neural network 90 will be described with reference to FIG. 5. FIG. 5 is a schematic diagram conceptually showing the structure of the neural network 90. As shown in FIG. 5, the neural network 90 has an encoder unit 91 as an “encoding unit” that extracts feature amounts (latent variables) TA from the input image data GA, and a decoder unit 92 as a “decoding unit” that reconstructs the image data GB from the feature amounts TA, and has the structure of a convolutional auto-encoder (CAE: Convolutional Auto-Encoder).

[0069] Since the structure of the convolutional auto-encoder is well-known, a detailed description will be omitted. The encoder unit 91 has a plurality of convolutional layers 93. In each convolutional layer 93, the result of performing a convolutional operation on the input data using a plurality of filters (kernels) 94 is output as the input data for the next layer. Similarly, the decoder unit 92 has a plurality of deconvolution layers 95. In each deconvolution layer 95, the result of performing a deconvolution operation on the input data using a plurality of filters (kernels) 96 is output as the input data for the next layer. And in the learning process described later, the weights (parameters) of each of the filters 94, 96 will be updated.

[0070] The inspection unit 78 is a functional unit that inspects the solder fillet 5. In the present embodiment, the inspection unit 78 inspects whether the solder fillet 5 is appropriately formed in terms of shape, area, quantity, etc.

[0071] The transport mechanism control unit 79 is a functional unit that drives and controls the transport mechanism 31, and controls the position of the printed circuit board 1 based on a command signal from the main control unit 71.

[0072] The memory unit 57 is composed of an HDD (Hard Disk Drive), an SSD (Solid State Drive), etc., and has a predetermined memory area for storing, for example, each AI model 101, 102 (the neural network 90 and the learning information acquired through its learning).

[0073] The communication unit 58 is equipped with, for example, a wireless communication interface conforming to communication standards such as wired LAN (Local Area Network) and wireless LAN, and is configured to be able to transmit and receive various data with the outside. For example, the results of inspections performed by the inspection unit 78 are output to the outside via the communication unit 58, or the results of inspections performed by the solder printing state inspection device 13 are input via the communication unit 58.

[0074] Next, the learning process of the neural network 90 performed by the solder fillet inspection device 16 will be described with reference to the flowchart of FIG. 6.

[0075] When the learning process is started based on the execution of a predetermined learning program, the main control unit 71 first performs preprocessing for learning the neural network 90 in step S101.

[0076] In this preprocessing, first, a large number of inspection information of the printed circuit board 1 previously stored in the solder fillet inspection device 16 is acquired. Subsequently, based on the inspection information, learning source image data Ig, which is image data related to the good solder fillets 5 that passed the post-reflow inspection, is acquired from the memory unit 57 (see, for example, FIG. 8).

[0077] This original image data Ig for learning pertains to the printed circuit board 1 after the reflow process and is used to obtain the following learning data Ga, Gb, Gc, Gd (hereinafter, may be abbreviated as "learning data Ga - Gd") for use in the learning of the neural network 90. Further, this original image data Ig for learning includes three - dimensional data, which is image data obtained by imaging the printed circuit board 1 with the camera 32d in a state where pattern light is irradiated from the first lighting device 32a or the second lighting device 32b, and two - dimensional data, which is image data obtained by imaging the printed circuit board 1 with the camera 32d in a state where uniform light is irradiated from the third lighting device 32c.

[0078] Note that the original image data Ig for learning may be image data that has not been subjected to any special processing as obtained by the camera 32d (for example, monochromatic luminance image data or RGB luminance image data), or may be image data obtained by performing a predetermined process on the image data obtained by the camera 32d (for example, HLS image data obtained by converting RGB image data, or height image data obtained by converting image data). Furthermore, the original image data Ig for learning may be image data obtained when using a so - called color highlight method (a method of irradiating the surface of the solder fillet 5 with red, green, and blue light at different incident angles and detecting the three - dimensional shape of the solder fillet 5 as two - dimensional hue information by photographing the reflected light of each color with the camera 32d).

[0079] Next, the first learning data Ga, Gb and the second learning data Gc, Gd are created from the acquired original image data Ig for learning (refer to FIGS. 11 - 14 for the learning data Ga - Gd). In this embodiment, the first learning data Ga, Gb correspond to the "learning data". The first learning data Ga, Gb are respectively used for the generation of the first AI model 101, and the second learning data Gc, Gd are used for the generation of the second AI model 102.

[0080] In obtaining each of the learning data Ga to Gd, first, the region occupied by the solder fillet 5 in the acquired original learning image data Ig is specified (see, for example, FIG. 9). When the original learning image data Ig is two-dimensional data, for example, the region occupied by the solder fillet 5 is specified using luminance, hue, saturation, etc. When the original learning image data Ig is three-dimensional data, for example, the region occupied by the solder fillet 5 is specified using height information, etc.

[0081] Next, an image of the connected component (block portion) in the region occupied by the specified solder fillet 5 is extracted as a single solder fillet image Ih (see, for example, FIG. 10). The single solder fillet image Ih corresponds to a single land 3. In this embodiment, among the regions occupied by the solder fillet 5, the connected component (block portion) located on the designed single land 3 is extracted as the single solder fillet image Ih. Incidentally, without considering the position of the land 3, simply extracting a single connected component (block portion) as the single solder fillet image Ih may be sufficient.

[0082] Next, an image frame for pasting the extracted single solder fillet image Ih is selected from the first image frame W1 and the second image frame W2 (see FIGS. 11 to 14 for each of the image frames W1 and W2).

[0083] In this embodiment, the first image frame W1 is a rectangular image with a height (the width in the vertical direction of the paper surface in FIG. 11 etc.) of n (pixels) and a width (the width in the horizontal direction of the paper surface in FIG. 11 etc.) of n (pixels), and its size (width and height) is set to be larger than the size of the extracted single solder fillet image Ih based on design data etc.

[0084] The second image frame W2 is a rectangular image with a height of m (pixels) and a width of m (pixels), and its size (width and height) is set to be smaller than the size of the first image frame W1.

[0085] In addition, each of the image frames W1 and W2 is assumed to have all pixels with the same value (when nothing is pasted). For example, the luminance, height information, etc. of each pixel constituting each of the image frames W1 and W2 are set to "0". It should be noted that this same value is preferably significantly different from the value of the constituent pixels of the single solder fillet image Ih. Therefore, it is preferable to use a value other than the commonly used value (for example, a negative number, etc.) as this same value.

[0086] The selection of the image frames W1 and W2 is performed based on the size and shape of the single solder fillet image Ih. When the size (width and height) of the single solder fillet image Ih is larger than the size of the second image frame W2, the first image frame W1 is selected. On the other hand, when the size of the single solder fillet image Ih is smaller than the size of the second image frame, the second image frame W2 is selected.

[0087] Next, by pasting the single solder fillet image Ih onto the selected image frames W1 and W2, each learning data Ga to Gd formed by providing the single solder fillet image Ih in the image frames W1 and W2 is obtained.

[0088] In this embodiment, the first learning data Ga (see, for example, FIG. 11) is obtained by pasting the single solder fillet image Ih corresponding to the solder fillet 5a onto the first image frame W1, and the first learning data Gb (see, for example, FIG. 12) is obtained by pasting the single solder fillet image Ih corresponding to the solder fillet 5b onto the first image frame W1.

[0089] Also, the second learning data Gc (see, for example, FIG. 13) is obtained by pasting the single solder fillet image Ih corresponding to the solder fillet 5c onto the second image frame W2, and the second learning data Gd (see, for example, FIG. 14) is obtained by pasting the single solder fillet image Ih corresponding to the solder fillet 5d onto the second image frame W2.

[0090] By performing adjustments to the attachment position and rotation processing of the solder fillet image Ih of 1, etc., in each learning data Ga to Gd, the center or centroid of the solder fillet image Ih of 1 coincides with the centers of the image frames W1 and W2, and the part on the electronic component 6 side in the solder fillet image Ih of 1 is set to face a predetermined direction. The "part on the electronic component 6 side in the solder fillet image Ih of 1" is a part adjacent to the electronic component 6 in the solder fillet image Ih of 1, and in this embodiment, it is linear. Then, the rotation processing of the solder fillet image Ih of 1 is performed so that this linear part faces a predetermined direction (for example, the right direction).

[0091] Then, by repeatedly performing the above processing such as extraction of the solder fillet image Ih of 1 and attachment of the solder fillet image Ih of 1 to the selected image frames W1 and W2, each learning data Ga to Gd is obtained from the original image data Ig for learning of 1. Furthermore, by using a plurality of original image data Ig for learning, finally the required number of first learning data Ga, Gb and second learning data Gc, Gd are obtained respectively. In this embodiment, each learning data Ga to Gd includes those obtained based on two-dimensional data and those obtained based on three-dimensional data.

[0092] When the required number of learning data Ga to Gd for learning is obtained in step S101, in the subsequent step S102, based on a command from the main control unit 71, the learning unit 77 prepares an unlearned neural network 90. For example, the neural network 90 stored in the storage unit 57 or the like in advance is read out. Or, based on the network configuration information (such as the number of layers of the neural network and the number of nodes in each layer) stored in the storage unit 57 or the like, the neural network 90 is constructed.

[0093] In this embodiment, as the neural network 90, one for performing learning using the first training data Ga and Gb and one for performing learning using the second training data Gc and Gd are separately constructed. Further, as the neural network 90, one for performing learning using the training data Ga to Gd obtained based on two-dimensional data and one for performing learning using the training data Ga to Gd obtained based on three-dimensional data are separately constructed. Therefore, in this embodiment, a total of four neural networks 90 are constructed.

[0094] In step S103, reconstructed image data is acquired. That is, based on a command from the main control unit 71, the learning unit 77 supplies the training data Ga to Gd acquired in step S102 as input data to the input layer of the neural network 90, and thereby acquires the reconstructed image data output from the output layer of the neural network 90. More specifically, the learning unit 77 supplies, as input data, the training data Ga to Gd corresponding to the neural network 90 among the training data Ga to Gd acquired in step S102 to the input layer of the neural network 90, and thereby acquires the reconstructed image data output from the output layer of the neural network 90. For example, the learning unit 77 supplies the first training data Ga and Gb obtained from two-dimensional data as input data to the input layer of the neural network 90 that performs learning using the first training data Ga and Gb, and acquires the reconstructed image data output from the neural network 90. That is, the learning unit 77 inputs appropriate training data Ga to Gd to each of the four types of neural networks 90 and acquires the output reconstructed image data.

[0095] In the subsequent step S104, the learning unit 77 compares the input training data Ga to Gd with the reconstructed image data output by the neural network 90, and determines whether the error is sufficiently small (whether it is equal to or less than a predetermined threshold).

[0096] Here, when the error is sufficiently small, in step S106, the learning unit 77 determines whether the end condition of learning is satisfied. For example, when an affirmative determination is made in step S104 without going through the process of step S105 described later for a predetermined number of consecutive times, or when learning using all of the prepared learning data Ga to Gd is repeated a predetermined number of times, it is determined that the end condition is satisfied. When the end condition is satisfied, the neural network 90 and its learning information (updated parameters and the like described later) are stored in the storage unit 57 as the AI models 101 and 102, and this learning process is terminated.

[0097] In the present embodiment, finally, as the first AI model 101, an AI model obtained by learning the first learning data Ga and Gb acquired from two-dimensional data and an AI model obtained by learning the first learning data Ga and Gb acquired from three-dimensional data are stored.

[0098] Also, as the second AI model 102, an AI model obtained by learning the second learning data Gc and Gd acquired from two-dimensional data and an AI model obtained by learning the second learning data Gc and Gd acquired from three-dimensional data are stored.

[0099] On the other hand, when the end condition is not satisfied in step S106, the process returns to step S102, and the learning of the neural network 90 is performed again.

[0100] Also, in step S104, when the error is not sufficiently small, after performing network update processing (learning of the neural network 90) in step S105, the process returns to step S103 again, and the above series of processes is repeated.

[0101] Specifically, in the network update process of step S105, known learning algorithms such as the Backpropagation algorithm are used to update the weights (parameters) of each of the filters 94 and 96 in the neural network 90 so that the loss function representing the difference between the training data Ga to Gd and the reconstructed image data becomes as small as possible. Note that as the loss function, for example, BCE (Binary Cross - entropy) can be used.

[0102] By repeating the processes of steps S103 to S105 many times, in the neural network 90, the error between the training data Ga to Gd and the reconstructed image data becomes as small as possible, and more accurate reconstructed image data is output.

[0103] And finally, each of the obtained AI models 101 and 102 generates reconstructed image data that almost matches the input image data when the image data related to the good solder fillet 5 is input. Also, when the image data related to the defective solder fillet 5 is input in terms of shape, area, and quantity, each of the AI models 101 and 102 generates reconstructed image data that almost matches the image data obtained by correcting the shape, area, and quantity of the solder fillet 5. That is, when the solder fillet 5 is defective, virtual image data related to the solder fillet 5 assuming no defective part is generated as the reconstructed image data related to the solder fillet 5.

[0104] Next, the inspection process performed by the solder fillet inspection device 16 will be described with reference to the flowchart of FIG. 7. However, the inspection process shown in FIG. 7 is a process executed for each inspection area on the printed circuit board 1.

[0105] When the printed circuit board 1 is carried into the solder fillet inspection device 16 and positioned at a predetermined inspection position, the inspection process is started based on the execution of a predetermined inspection program.

[0106] When the inspection process starts, first, in step S301, an image data acquisition process is performed. In the image data acquisition process, inspection original image data Ik (for example, see FIG. 15) related to the printed circuit board 1 to be inspected is acquired. The inspection original image data Ik is image data for obtaining inspection image data Ka, Kb, Kc, Kd (hereinafter, may be abbreviated as "inspection image data Ka to Kd") described later. In addition, in the present embodiment, as the printed circuit board 1 to be inspected, an example is given in which the solder fillet 5 protrudes from the land 3 or there are abnormalities in the shape or area of the solder fillet 5.

[0107] The inspection original image data Ik includes three-dimensional data, which is image data obtained by imaging the printed circuit board 1 with the camera 32d in a state where pattern light is irradiated from the first illumination device 32a or the second illumination device 32b, and two-dimensional data, which is image data obtained by imaging the printed circuit board 1 with the camera 32d in a state where uniform light is irradiated from the third illumination device 32c. In the image data acquisition process, a three-dimensional data acquisition process and a two-dimensional data acquisition process are performed.

[0108] First, the three-dimensional data acquisition process will be described. In this process, while changing the phase of the first pattern light irradiated from the first illumination device 32a, four imaging processes are performed under the first pattern light with different phases, and then while changing the phase of the second pattern light irradiated from the second illumination device 32b, four imaging processes are performed under the second pattern light with different phases, and a total of eight sets of three-dimensional data are acquired. This will be described in detail below.

[0109] As described above, when the printed circuit board 1 carried into the solder fillet inspection device 16 is positioned and fixed at a predetermined inspection position, based on a command from the main control unit 71, the movement mechanism control unit 76 first drives and controls the X-axis movement mechanism 32e and the Y-axis movement mechanism 32f to move the inspection unit 32, and adjusts the imaging field (imaging range) of the camera 32d to a predetermined inspection area of the printed circuit board 1.

[0110] At the same time, the lighting control unit 72 switches and controls the liquid crystal shutters 32a2 and 32b2 of both lighting devices 32a and 32b, and sets the positions of the first grid and the second grid formed on the both liquid crystal shutters 32a2 and 32b2 to a predetermined reference position.

[0111] When the switching setting of the first grid and the second grid is completed, the lighting control unit 72 causes the first light source 32a1 of the first lighting device 32a to emit light and irradiates the first pattern light, and the camera control unit 73 drives and controls the camera 32d to execute the first imaging process under the first pattern light. The image data generated by the imaging process is taken into the image acquisition unit 74 at any time (the same applies hereinafter). Thereby, three-dimensional data of an inspection area including a plurality of lands 3 (solder fillets 5) is acquired.

[0112] Thereafter, simultaneously with the end of the first imaging process under the first pattern light, the lighting control unit 72 turns off the first light source 32a1 of the first lighting device 32a and executes the switching process of the first liquid crystal shutter 32a2. Specifically, the position of the first grid formed on the first liquid crystal shutter 32a2 is switched and set from the reference position to a second position where the phase of the first pattern light is shifted by a quarter pitch (90°).

[0113] When the switching setting of the first grid is completed, the lighting control unit 72 causes the light source 32a1 of the first lighting device 32a to emit light and irradiates the first pattern light, and the camera control unit 73 drives and controls the camera 32d to execute the second imaging process under the first pattern light. Thereafter, by repeating the same process, four types of three-dimensional data under the first pattern lights with different phases by 90° each are acquired.

[0114] Subsequently, the lighting control unit 72 causes the second light source 32b1 of the second lighting device 32b to emit light and irradiates the second pattern light, and the camera control unit 73 drives and controls the camera 32d to execute the first imaging process under the second pattern light.

[0115] Thereafter, upon completion of the first imaging process under the second pattern light, the illumination control unit 72 turns off the second light source 32b1 of the second illumination device 32b and executes a switching process for the second liquid crystal shutter 32b2. Specifically, the position of the second grating formed on the second liquid crystal shutter 32b2 is switched and set from the reference position to a second position where the phase of the second pattern light is shifted by a quarter pitch (90°).

[0116] When the switching setting of the second grating is completed, the illumination control unit 72 turns on the light source 32b1 of the second illumination device 32b to irradiate the second pattern light, and the camera control unit 73 drives and controls the camera 32d to execute the second imaging process under the second pattern light. Thereafter, by repeating the same process, four sets of three-dimensional data under the second pattern light with different phases by 90° each are acquired.

[0117] Next, the process of acquiring two-dimensional data will be described. In this process, based on a command from the main control unit 71, the illumination control unit 72 turns on the third illumination device 32c to irradiate uniform light onto a predetermined inspection area, and the camera control unit 73 drives and controls the camera 32d to execute an imaging process under the uniform light. As a result, a predetermined inspection area on the printed circuit board 1 is imaged, and two-dimensional data related to the inspection area is acquired.

[0118] The acquired original inspection image data Ik (three-dimensional data and two-dimensional data) is stored in the storage unit 57.

[0119] Note that the original inspection image data Ik may be image data that has not been subjected to any special processing as obtained by the camera 32d (for example, monochromatic luminance image data or RGB luminance image data), or may be image data obtained by subjecting the image data obtained by the camera 32d to a predetermined process (for example, HLS image data obtained by converting RGB image data or height image data obtained by converting image data). Further, the original inspection image data Ik may be image data acquired when using a so-called color highlight method.

[0120] Next, in step S302, an inspection image data acquisition process is executed. In the inspection image data acquisition process, based on the inspection original image data Ik obtained in the image data acquisition process, first inspection image data Ka, Kb and second inspection image data Kc, Kd, which will be described later, are acquired respectively.

[0121] In obtaining these inspection image data Ka to Kd, first, the area occupied by the solder fillet 5 in the acquired inspection original image data Ik is specified (see, for example, FIG. 16). The area occupied by the solder fillet 5 can be specified using luminance, hue, saturation, height information, etc.

[0122] Next, a single solder fillet image Ih (see, for example, FIGS. 17 and 18) in the specified area occupied by the solder fillet 5 is extracted. The single solder fillet image Ih corresponds to the single land 3. In the present embodiment, among the connected components (block portions) in the specified area, the entire image of the one whose at least a part overlaps with the single land 3 on the design data or manufacturing data is extracted as the single solder fillet image Ih. Therefore, when a part of the connected component protrudes from the land 3 on the design data or manufacturing data, the single solder fillet image Ih is composed of the entire connected component including the protruding part from the land 3 (see, for example, FIG. 18). Note that, without using design data or the like, a single connected component may be simply extracted as the single solder fillet image Ih. Also, the single solder fillet image Ih may include the land 3 and the electrode 6a located around it in addition to the connected component (block portion) of the solder fillet 5. In the present embodiment, an inspection unit 78 that extracts the single solder fillet image Ih from the inspection original image data Ik constitutes "solder fillet image extraction means".

[0123] Then, based on the size and shape of the extracted single solder fillet image Ih, appropriate image frames are selected from the first image frame W1 and the second image frame W2, and inspection image data Ka to Kd, which are formed by providing the single solder fillet image Ih to the selected image frames W1, W2, are acquired (see FIGS. 19 to 24 for the inspection image data Ka to Kd).

[0124] That is, when the size of the extracted solder fillet image Ih of 1 is larger than the size of the second image frame W2, the solder fillet image Ih of 1 is pasted onto the first image frame W1 to obtain the first inspection image data Ka, Kb (see, for example, FIGS. 19 to 21). Therefore, the sizes (width and height) of the first inspection image data Ka, Kb are the same as the sizes of the first learning data Ga, Gb. In FIG. 20, for reference, the solder fillet 5 with a normal shape and area is virtually shown by a two-dot chain line.

[0125] Also, when the size of the extracted solder fillet image Ih of 1 is smaller than the size of the second image frame W2, the solder fillet image Ih of 1 is pasted onto the second image frame W2 to obtain the second inspection image data Kc, Kd (see, for example, FIGS. 22 to 24). Therefore, the sizes of the second inspection image data Kc, Kd are the same as the sizes of the second learning data Gc, Gd.

[0126] In addition, when pasting the solder fillet image Ih of 1 onto the image frames W1, W2, adjustments of the pasting position and rotation processing of the image are performed. As a result, for each inspection image data Ka to Kd, similar to each learning data Ga to Gd, the center or centroid of the solder fillet image Ih of 1 coincides with the center of the image frames W1, W2, and the part on the electronic component 6 side in the solder fillet image Ih of 1 faces a predetermined direction.

[0127] Then, by repeatedly performing the above processes such as extraction of the solder fillet image Ih of 1 and pasting of the solder fillet image Ih of 1 onto the selected image frames W1, W2, each inspection image data Ka to Kd is obtained from the original inspection image data Ik of 1. In addition, in the present embodiment, each inspection image data Ka to Kd includes those obtained based on two-dimensional data and those obtained based on three-dimensional data. In the present embodiment, the inspection unit 78 that obtains the inspection image data Ka to Kd constitutes the "inspection image data acquisition means".

[0128] In the subsequent step S303, a reconstructed image data acquisition process is executed. Specifically, based on a command from the main control unit 71, the inspection unit 78 inputs the inspection image data Ka to Kd acquired in step S302 into the input layers of the AI models 101 and 102 corresponding to the types of the inspection image data Ka to Kd. Therefore, the first inspection image data Ka and Kb are input into the first AI model 101, and the second inspection image data Kc and Kd are input into the second AI model 102, respectively. Also, the inspection image data Ka to Kd acquired based on two-dimensional data are input into the AI models 101 and 102 corresponding to the two-dimensional data, and the inspection image data Ka to Kd acquired based on three-dimensional data are input into the AI models 101 and 102 corresponding to the three-dimensional data. Then, the image data reconstructed by the AI models 101 and 102 and output from the output layer is acquired as the reconstructed image data. The acquired reconstructed image data is stored in association with the inspection image data Ka to Kd that is the source of the reconstructed image data.

[0129] Here, when the inspection image data Ka and Kc (see FIGS. 20 and 22) related to the solder fillet 5 with inappropriate shape or the like are input into each of the AI models 101 and 102, due to the learning as described above, as the reconstructed image data S, image data related to the good solder fillet 5 with the shape or the like corrected is output (for example, see FIGS. 25 and 26).

[0130] On the other hand, when the inspection image data Ka to Kd related to the good solder fillet 5 are input into each of the AI models 101 and 102, as the reconstructed image data S, image data related to the good solder fillet 5 that is substantially the same as the inspection image data Ka to Kd is output. Note that the size (width and height) of the reconstructed image data S is the same as the size of the inspection image data Ka to Kd that is the source thereof. In the present embodiment, the inspection unit 78 that acquires the reconstructed image data S constitutes the "reconstructed image data acquisition means".

[0131] In step S304, a pass / fail determination process based on the acquired reconstructed image data S is performed. In the pass / fail determination process, based on a command from the main control unit 71, the inspection unit 78 compares the entire inspection image data Ka to Kd acquired in step S302 above with the entire reconstructed image data S acquired in step S303 using the inspection image data Ka to Kd, and calculates the difference between the two image data Ka to Kd, S. For example, by comparing the dots (pixels) at the same coordinates in both image data Ka to Kd, S, the area (number of dots) of a block of dots where the luminance difference is equal to or greater than a predetermined value is calculated. In the present embodiment, the inspection unit 78 that compares the inspection image data Kd to Kd and the reconstructed image data S constitutes "comparison means". Also, the process of comparing the inspection image data Ka to Kd and the reconstructed image data S corresponds to the "comparison process".

[0132] Subsequently, the inspection unit 78 determines whether the calculated difference is greater than a predetermined threshold. Then, when the calculated difference is greater than the predetermined threshold, the inspection unit 78 determines "good product", while when the difference is less than the predetermined threshold, it determines "defective product".

[0133] Furthermore, the inspection unit 78 performs the above determination for all the inspection image data Ka to Kd related to the inspection area of the printed circuit board 1. When it is determined "good product" for all the inspection image data Ka to Kd, it determines "good product" for the inspection area and stores this result in the storage unit 57. On the other hand, when, as a result of performing the above determination for all the inspection image data Ka to Kd related to the inspection area, it is determined "defective product" for at least one of the inspection image data Ka to Kd, it determines "defective product" for the inspection area and stores this result in the storage unit 57.

[0134] Then, as a result of performing the above inspection process for all the inspection areas on the printed circuit board 1, when it is determined "good product" for all the inspected areas, the solder fillet inspection device 16 determines that the printed circuit board 1 has no abnormality in the solder fillets 5 (pass determination) and stores this result in the storage unit 57.

[0135] On the other hand, if there is even one inspected area determined as a "defective product", the solder fillet inspection device 16 determines that the printed circuit board 1 has an abnormality in the solder fillet 5 (failure determination), stores this result in the storage unit 57, and notifies the outside of this fact via the display unit 56, the communication unit 58, etc.

[0136] As described in detail above, according to this embodiment, the inspection image data Ka to Kd are provided with one solder fillet image Ih in the image frames W1 and W2. Therefore, the sizes (width and height) of the respective inspection image data Ka to Kd do not vary finely depending on the size of the land 3 and are constant. As a result, it is not necessary to prepare a large number of different AI models (identifying means) for each size of the land 3, and the labor and effort required to obtain the AI models 101 and 102 can be reduced. Further, even when the sizes of the lands 3 are different, the AI models 101 and 102 can be used in common.

[0137] Furthermore, the image frames W1 and W2 of the learning data Ga to Gd and the image frames W1 and W2 of the inspection image data Ka to Kd are the same size, and the sizes of the learning data Ga to Gd and the inspection image data Ka to Kd are the same. Therefore, when the inspection image data Ka to Kd are input to the AI models 101 and 102, appropriate reconstructed image data S corresponding to the inspection image data Ka to Kd can be more reliably output, and thus the quality determination of the solder fillet 5 can be performed more accurately. As a result, good inspection accuracy can be obtained more reliably.

[0138] In addition, the inspection image data Ka to Kd are compared with the reconstructed image data S obtained by inputting the inspection image data Ka to Kd into the AI models 101 and 102, and based on the comparison result, the quality of the solder fillet 5 is determined. Therefore, the two image data Ka to Kd and S to be compared are each related to the same solder fillet 5. Accordingly, unlike the method of determining the quality by comparison with a separately prepared standard, it is not necessary to set relatively loose inspection conditions to prevent false detection, and relatively strict inspection conditions can be set. Furthermore, in the two image data Ka to Kd and S to be compared, the imaging conditions of the printed circuit board 1 to be inspected (for example, the placement position, placement angle, deflection, etc. of the printed circuit board 1) and the imaging conditions on the inspection apparatus 16 side (for example, the illumination state, the angle of view of the camera, etc.) can be made to coincide. These factors combined enable the quality determination of the solder fillet 5 to be performed with higher accuracy.

[0139] Also, in the present embodiment, one solder fillet image Ih constituting the inspection image data Ka to Kd includes not only the image of the portion located on the land 3 in the solder fillet 5 but also the image of the portion protruding from the land 3 in the solder fillet 5. Thereby, it becomes possible to appropriately perform the quality determination regarding the solder fillet 5 with a part protruding from the land 3, and the inspection accuracy can be further enhanced.

[0140] Furthermore, in the present embodiment, when the size of one solder fillet image Ih is relatively small, inspection image data Kc and Kd of a relatively small size are obtained by providing one solder fillet image Ih in a second image frame W2 of a relatively small size. Then, by inputting this relatively small inspection image data Kc and Kd into the second AI model 102, reconstructed image data S is output, and the relatively small inspection image data Kc and Kd and the reconstructed image data S are respectively compared. Therefore, compared with the case where the image frame of the inspection image data is always of a constant size, the processing for obtaining the reconstructed image data S and the comparison processing by the inspection unit 78 can be speeded up, and thus the inspection speed can be further improved.

[0141] In addition, since the orientations and positions of the solder fillets 5 in the learning data Ga to Gd and the inspection image data Ka to Kd are generally aligned, even if the learning data Ga to Gd used for generating the AI models 101 and 102 is relatively small, it is possible to accurately determine the quality of the solder fillets 5. That is, it is possible to obtain good inspection accuracy while more effectively reducing the labor and effort required to obtain the AI models 101 and 102.

[0142] Note that the present invention is not limited to the description of the above embodiment, and for example, it may be implemented as follows. Of course, other application examples and modification examples not illustrated below are also naturally possible.

[0143] (a) In the above embodiment, in the pass / fail determination step of step S304, the entire inspection image data Ka to Kd and the entire reconstructed image data S are configured to be compared.

[0144] On the other hand, only one solder fillet image Ih in the inspection image data Ka to Kd may be set as a comparison target, and the inspection image data Ka to Kd and the reconstructed image data S may be compared. That is, one solder fillet image Ih in the inspection image data Ka to Kd and a region of the reconstructed image data S that overlaps with the one solder fillet image Ih may be configured to be compared.

[0145] When configured in this way, since the portions other than the one solder fillet image Ih in the inspection image data Ka to Kd are not comparison targets, the processing load related to the comparison of the two image data Ka to Kd and S can be reduced compared to the case where the entire two image data Ka to Kd and S are compared. Therefore, the inspection can be speeded up and made more efficient. In addition, it is possible to more reliably prevent the portions other than the one solder fillet image Ih in the inspection image data Ka to Kd, that is, the portions unrelated to the solder fillet 5, from affecting the pass / fail determination, and thus it is possible to further improve the inspection accuracy.

[0146] It may also be configured to compare the inspection image data Ka to Kd and the reconstructed image data S by using only the region related to the solder fillet 5 in the reconstructed image data S as a comparison target. That is, it may be configured to compare the region related to the solder fillet 5 in the reconstructed image data S with the overlapping region of the region related to the solder fillet 5 in the inspection image data Ka to Kd. Of course, the above two comparison methods may be used in combination.

[0147] (b) In the above embodiment, in the learning of the neural network 90, the learning data Ga to Gd are acquired by using the original learning image data Ig related to the printed circuit board 1 that passed the post-reflow inspection. On the other hand, for example, the learning data Ga to Gd may be acquired by using the original learning image data related to the good solder fillets 5 visually selected by an operator after the reflow process.

[0148] Also, the learning unit 77 may acquire the learning data Ga to Gd by using the image data of the virtual good solder fillets 5.

[0149] (c) In the above embodiment, as the AI models 101 and 102, those corresponding to two-dimensional data and those corresponding to three-dimensional data are provided separately, but it may be configured to provide a common AI model corresponding to each of the two-dimensional data and the three-dimensional data.

[0150] Furthermore, it may be configured to omit the second AI model 102. In this case, the sizes of the second inspection image data Kc and Kd may be made the same as the sizes of the first inspection image data Ka and Kd, and the first AI model 101 may be configured to perform an inspection based on the inspection image data Ka to Kd.

[0151] (d) The configurations and learning methods of the AI models 101 and 102 (neural network 90) are not limited to the above-described embodiments. For example, when performing learning processing of the neural network 90, the process of acquiring reconstructed image data, etc., a configuration may be adopted in which various data are subjected to processing such as normalization as necessary. Further, the structure of the neural network 90 is not limited to that shown in FIG. 5, and for example, a configuration in which a pooling layer is provided after the convolutional layer 93 may be adopted. Of course, configurations may be adopted in which the number of layers of the neural network 90, the number of nodes in each layer, the connection structure of each node, etc. are different.

[0152] Furthermore, in the above-described embodiment, the AI models 101 and 102 (neural network 90) are generative models having the structure of a convolutional autoencoder (CAE), but the present invention is not limited thereto, and for example, a generative model having the structure of a different type of autoencoder such as a variational autoencoder (VAE) may be adopted.

[0153] Also, in the above-described embodiment, the neural network 90 is configured to be learned by the error backpropagation method, but the present invention is not limited thereto, and a configuration may be adopted in which learning is performed using various other learning algorithms.

[0154] In addition, the neural network 90 may be configured by an AI processing dedicated circuit such as a so-called AI chip. In that case, only learning information such as parameters may be stored in the storage unit 57, and the AI processing dedicated circuit reads this and sets it in the neural network 90, whereby the AI models 101 and 102 may be configured.

[0155] In addition, in the above-described embodiment, the control device 33 includes a learning unit 77 and is configured to perform learning of the neural network 90 within the control device 33. However, the present invention is not limited thereto. For example, the learning unit 77 may be omitted, and the learning of the neural network 90 may be performed outside the control device 33. The AI models 101 and 102 (the learned neural network 90) that have been learned externally may be stored in the storage unit 57.

[0156] (e) In the above-described embodiment, the inspection original image data Ik is acquired as two-dimensional data and three-dimensional data. However, a configuration may be adopted in which only one of the two-dimensional data and the three-dimensional data is acquired. Further, in accordance with the data to be acquired, as the AI models 101 and 102, only those corresponding to one of the two-dimensional data and the three-dimensional data may be provided.

[0157] (f) In the above-described embodiment, the inspection image data Ka to Kd is acquired by attaching one solder fillet image Ih to the image frames W1 and W2. In contrast, the inspection image data Ka to Kd may be acquired as follows. That is, first, when extracting the image of the connected component (block portion) in the region occupied by the identified solder fillet 5 as one solder fillet image Ih, an extracted image having the same size as the image frames W1 and W2 is obtained by extracting one solder fillet image Ih and its peripheral portion. Then, the inspection image data Ka to Kd may be acquired by replacing the values of each pixel in the peripheral portion of the extracted image with the same value (for example, setting the luminance or height to "0"). Of course, the learning data Ga to Gd may be acquired using a similar method.

Description of Reference Numerals

[0158] 1…Printed circuit board, 3…Land, 5…Solder fillet, 6…Electronic component, 16…Solder fillet inspection device, 32d…Camera (image data acquisition means), 78…Inspection unit (inspection image data acquisition means, reconstructed image data acquisition means, comparison means, solder fillet image extraction means), 90…Neural network, 91…Encoder unit (encoding unit), 92…Decoder unit (decoding unit), 101…First AI model (identification means), 102…Second AI model (second identification means) Ih…Solder fillet image of 1, W1…First image frame (image frame), W2…Second image frame.

Claims

1. A solder fillet inspection apparatus for inspecting a solder fillet for soldering an electronic component on a printed circuit board, image data acquisition means capable of acquiring image data of a predetermined inspection area on the printed circuit board including the solder fillet, an encoding unit that extracts feature amounts from the input image data, and an identification means generated by learning only the image data related to a good solder fillet as learning data for a neural network having a decoding unit that reconstructs the image data from the feature amounts, inspection image data acquisition means for acquiring inspection image data including an image of a solder fillet to be inspected based on the image data acquired by the image data acquisition means, reconstructed image data acquisition means capable of acquiring, as reconstructed image data, the image data reconstructed by inputting the inspection image data to the identification means, comparison means capable of comparing the inspection image data and the reconstructed image data, configured to be able to determine the quality of the solder fillet based on the comparison result by the comparison means, the learning data is formed by providing an image of one solder fillet indicating a solder fillet corresponding to one land in an image frame having a size larger than the size of the image of the one solder fillet, the inspection image data acquisition means acquires the inspection image data of the same size as the learning data, in which the image of the one solder fillet extracted from the image data acquired by the image data acquisition means is provided in an image frame of the same size as the image frame of the learning data. The solder fillet inspection apparatus according to claim 1, characterized in that.

2. comprising solder fillet image extraction means for extracting the image of the one solder fillet constituting the inspection image data from the image data acquired by the image data acquisition means, the solder fillet image extraction means is capable of specifying the area occupied by the solder fillet in the image data acquired by the image data acquisition means, and extracting, as the image of the one solder fillet constituting the inspection image data, the image of the connected component in the specified area. The solder fillet inspection apparatus according to claim 1, characterized in that.

3. For a neural network having an encoding unit that extracts feature amounts from input image data and a decoding unit that reconstructs image data from the feature amounts, it is provided with second discrimination means generated by learning only image data related to good solder fillets as second learning data. The second learning data is formed by providing the solder fillet image of 1 in a second image frame having a size larger than the size of the solder fillet image of 1 and smaller than the size of the image frame of the learning data. When the size of the solder fillet image of 1 extracted from the image data acquired by the image data acquisition means is smaller than the size of the second image frame, the inspection image data acquisition means acquires the inspection image data of the same size as the second learning data, which is formed by providing the solder fillet image of 1 in the second image frame. The reconstructed image data acquisition means acquires the reconstructed image data reconstructed by inputting the inspection image data to the second discrimination means. The comparison means is configured to compare the inspection image data and the reconstructed image data. The solder fillet inspection apparatus according to claim 1, characterized in that.

4. The learning data and the inspection image data are set such that the center or centroid of the solder fillet image of 1 coincides with the center of the image frame, and the part on the electronic component side in the solder fillet image of 1 faces a predetermined direction. The solder fillet inspection apparatus according to claim 1, characterized in that.

5. The comparison means is configured to be able to compare the inspection image data and the reconstructed image data by using only the solder fillet image of 1 in the inspection image data as a comparison target. The solder fillet inspection apparatus according to claim 1, characterized in that.

6. A solder fillet inspection method for inspecting a solder fillet for soldering an electronic component on a printed circuit board, An image data acquisition step capable of acquiring image data of a predetermined inspection region in the printed circuit board including the solder fillet, An inspection image data acquisition step of acquiring inspection image data including an image of a solder fillet to be inspected based on the image data acquired in the image data acquisition step. For a neural network having an encoding unit that extracts feature amounts from input image data and a decoding unit that reconstructs image data from the feature amounts, using discrimination means generated by learning only image data related to good solder fillets as learning data, a reconstructed image data acquisition step capable of acquiring, as reconstructed image data, image data reconstructed by inputting the inspection image data acquired in the inspection image data acquisition step into the discrimination means, A comparison step of comparing the inspection image data and the reconstructed image data, Based on the comparison result in the comparison step, determining the quality of the solder fillet, The learning data is formed by providing a single solder fillet image showing a solder fillet corresponding to a single land in an image frame having a size larger than the size of the single solder fillet image. In the inspection image data acquisition step, the inspection image data of the same size as the learning data, which is formed by providing the single solder fillet image extracted from the image data acquired in the image data acquisition step in an image frame having the same size as the image frame of the learning data, is acquired. A solder fillet inspection method characterized by this.

Images