Solder inspection device

Abstract

To provide a solder inspection device that can perform highly accurate inspections while taking into account the direction of movement of the squeegee. [Solution] An AI model 101 is generated by training a predetermined neural network using only image data related to good solder paste as training data. Inspection image data is input to this AI model 101 to obtain reconstructed image data, and the quality of the solder paste is determined by comparing the inspection image data and the reconstructed image data. The AI ​​model 101 is generated by training only image data related to solder paste with a common squeegee movement direction as training data. The inspection unit 79 inputs the inspection image data to the AI ​​model 101, where the squeegee movement direction in the trained training data is the same as the squeegee movement direction related to the solder paste to be inspected included in the inspection image data.

Description

Technical Field

[0001] The present invention relates to a solder inspection apparatus for inspecting cream solder printed on a substrate.

Background Art

[0002] Generally, in a substrate manufacturing line for mounting electronic components on a printed circuit board, first, cream solder is printed on the lands of the printed circuit board (solder printing process). Next, electronic components are temporarily fixed on the printed circuit board based on the viscosity of the cream solder (mounting process). Then, such a printed circuit board is led into a reflow furnace, and soldering is performed by heating and melting the cream solder (reflow process).

[0003] In the solder printing process, first, with the lower surface of a predetermined mask (for example, a screen mask or a metal mask) in contact with the printed circuit board, cream solder is supplied onto the upper surface of the mask. The mask has a plurality of openings corresponding to the respective lands of the printed circuit board. After the supply of the cream solder, the cream solder is filled into the openings by moving a predetermined squeegee while contacting the upper surface of the mask. Then, by separating the printed circuit board from the lower surface of the mask, cream solder is printed on the lands of the printed circuit board.

[0004] Also, in a substrate manufacturing line as described above, an inspection apparatus for inspecting the printed circuit board may be provided. Recently, as an inspection apparatus for inspecting a printed circuit board, an apparatus using an AI model has been proposed. As an inspection apparatus using an AI model, for example, an apparatus that inspects the presence or absence of foreign matter on a printed circuit board by comparing inspection image data (original image data) of an inspection area on the printed circuit board with reconstructed image data generated by inputting the inspection image data into an AI model (identification means) is known (for example, see Patent Document 1, etc.).

[0005] Furthermore, in the inspection of printed circuit boards, solder paste is sometimes inspected by determining, for example, its shape (two-dimensional or three-dimensional shape) and whether or not foreign matter is attached to it. Therefore, it is conceivable to perform solder paste inspection using an AI model like the one described above. In this case, in order to obtain the AI ​​model, it is necessary to train it using image data related to solder paste as training data. [Prior art documents] [Patent Documents]

[0006] [Patent Document 1] Japanese Patent Publication No. 2022-61456 [Overview of the Initiative] [Problems that the invention aims to solve]

[0007] Incidentally, the direction of movement of the squeegee when printing solder paste is not necessarily the same even for the same type of printed circuit board. For example, even with the same type of printed circuit board, some solder paste may have been printed by the squeegee moving in the forward direction (e.g., from right to left), while others may have been printed by the squeegee moving in the reverse direction (e.g., from left to right). Therefore, if training data related to solder paste with different squeegee movement directions is mixed together and fed into an AI model, the AI ​​model may not be able to properly learn the differences in solder paste shape tendencies depending on the squeegee movement direction. As a result, the accuracy of the reconstructed image data generated by the AI ​​model may decrease, and consequently, the inspection capability may not be sufficiently improved.

[0008] This invention has been made in view of the above circumstances, and its purpose is to provide a solder inspection device that can perform accurate inspections taking into account the direction of movement of the squeegee. [Means for solving the problem]

[0009] Below, we will describe, in separate sections, each means suitable for achieving the above objectives. Furthermore, we will add notes on the effects and benefits specific to each means as needed.

[0010] Method 1. A solder inspection apparatus for inspecting solder paste printed on a printed circuit board, comprising supplying solder paste to the upper surface of a mask having an opening, moving a predetermined squeegee in contact with the upper surface of the mask, and filling the opening with solder paste, Image data acquisition means capable of acquiring image data of a predetermined area to be inspected on the printed circuit board including printed solder paste, An identification means generated by training a neural network having an encoding unit that extracts features from input image data and a decoding unit that reconstructs image data from the features, using only image data related to good quality solder paste as training data, An inspection image data acquisition means that acquires inspection image data, including an image of the solder paste to be inspected, based on the image data acquired by the aforementioned image data acquisition means, A reconstructed image data acquisition means capable of inputting the aforementioned inspection image data into the identification means and acquiring the reconstructed image data as reconstructed image data, The system includes a comparison means capable of comparing the aforementioned inspection image data and the aforementioned reconstructed image data, Based on the comparison results obtained by the aforementioned comparison means, the system is configured to determine the quality of the solder paste. The aforementioned identification means is generated by training only with image data relating to solder paste where the direction of movement of the squeegee is common, Solder inspection apparatus characterized in that the reconstructed image data acquisition means is configured to input the inspection image data to the identification means such that the direction of movement of the squeegee in the learned learning data is the same as the direction of movement of the squeegee related to the solder paste to be inspected included in the inspection image data.

[0011] Furthermore, the training data may be generated from image data (actual image data) obtained by imaging a printed circuit board with good solder paste printed on it (i.e., an image relating to actual solder paste), or it may be a virtually generated image relating to good solder paste. Examples of the actual image data include image data accumulated from previous inspections, and image data of good printed circuit boards that have been visually sorted by an operator after the solder paste has been printed.

[0012] Furthermore, the above-mentioned "neural network" includes, for example, a convolutional neural network having multiple convolutional layers. The above-mentioned "learning" includes, for example, deep learning. The above-mentioned "discrimination means (generative model)" includes, for example, an autoencoder or a convolutional autoencoder.

[0013] In addition, the "identification means" is generated by training it only with image data related to good solder paste. Therefore, when inspection image data related to defective solder paste is input to the identification means, the reconstructed image data generated will be almost identical to the inspection image data after the defective parts have been corrected (for example, foreign matter has been removed, or the shape and size have been corrected). In other words, when there are defective parts in the solder paste, the reconstructed image data related to the solder paste will be a virtual image data of the solder paste assuming that there are no defective parts.

[0014] Furthermore, the phrase "the direction of squeegee movement is common" includes not only cases where the direction of squeegee movement is exactly the same in multiple image data, but also cases where one component of the direction of squeegee movement is common. Therefore, if the direction of squeegee movement in one image data is from right to left, and the direction of squeegee movement in another image data is from upper right to lower left, then the right-to-left directional component is common in both image data, and thus the direction of squeegee movement in both image data can be said to be common.

[0015] According to the above-described means 1, the identification means is generated by learning only the learning data relating to solder paste where the squeegee movement direction is common. The reconstructed image data acquisition means inputs the inspection image data into the identification means, where the squeegee movement direction in the learned learning data is the same as the squeegee movement direction relating to the solder paste to be inspected included in the inspection image data. For example, if the squeegee movement direction relating to the solder paste to be inspected included in the inspection image data is from right to left, the reconstructed image data acquisition means inputs the inspection image data into the identification means, which has learned the learning data in which the squeegee movement direction is the same as the squeegee movement direction relating to the solder paste (for example, the squeegee movement direction is set to be from right to left). Therefore, the identification means into which the inspection image data is input is adjusted (tuned) to match the squeegee movement direction in that inspection image data. As a result, the accuracy of the reconstructed image data reconstructed when the inspection image data is input into the identification means can be improved, and superior inspection capabilities can be obtained more reliably.

[0016] Means 2. The learning data is obtained by setting one solder region image representing one solder paste corresponding to one land, such that the direction of movement of the squeegee related to the solder paste is a predetermined direction, and providing this direction common image in an image frame larger than the size of the one direction common image. The solder inspection apparatus according to means 1, characterized in that the inspection image data acquisition means acquires the inspection image data obtained by setting the solder region image of the image data acquired by the image data acquisition means such that the direction of movement of the squeegee related to the solder paste in the solder region image is the same direction as the direction of movement of the squeegee in the learning data, and providing the inspection image data in an image frame of the same size as the image frame of the learning data.

[0017] Furthermore, the solder region image of 1 may be an image showing the entire mass of solder paste located at least partially on land 1, or it may be an image showing only the portion of the solder paste mass that is located on land 1. In the former case, if the solder paste extends beyond land 1, the image will be an image of the entire solder paste including the protruding portion. On the other hand, in the latter case, if the solder paste extends beyond land 1, the image will be an image of a portion of the solder paste excluding the protruding portion. Also, the solder region image of 1 only needs to correspond to land 1, and may be an image consisting of a mass of 1, or an image consisting of multiple masses.

[0018] According to method 2 described above, the inspection image data is provided by setting a common orientation image within an image frame. Therefore, the size (width and height) of the inspection image data will match the size of the image frame and will remain constant without fluctuating finely depending on the size of the land (solder paste). This eliminates the need to prepare multiple different identification means for each land size (size of the inspection image data).

[0019] Furthermore, the training data consists of a common direction image obtained by setting the squeegee's movement direction to a predetermined direction and placing it in the image frame. Similarly, the inspection image data consists of a common direction image obtained by setting the squeegee's movement direction to be the same as the direction in the training data and placing it in the image frame. Consequently, the squeegee's movement direction in the inspection image data is consistent, and the identification means is adjusted (tuned) to match the squeegee's movement direction. This eliminates the need to prepare multiple different identification means for each squeegee movement direction in the inspection image data. As a result, combined with the fact that multiple different identification means do not need to be prepared for each land size as described above, the effort and time required to obtain the identification means can be reduced. In addition, the identification means can be used in common regardless of the land size or the squeegee's movement direction.

[0020] In addition, the image frame of the learning data and the image frame of the inspection image data are of the same size, and the sizes of the learning data and the inspection image data are the same. Therefore, when the inspection image data is input to the identification means, appropriate reconstructed image data corresponding to the inspection image data can be more reliably output. As a result, the inspection of cream solder can be performed more accurately.

[0021] Also, learning data with a common squeegee movement direction can be obtained from a plurality of solder region images with different squeegee movement directions for use in training one identification means. For example, when there are a solder region image with the squeegee moving from right to left and a solder region image with the squeegee moving from left to right, while using one solder region image as it is to obtain learning data, image processing (such as rotation or inversion) is performed on the other solder region image to align the squeegee movement direction, and thus a plurality of learning data with a common squeegee movement direction can be obtained. Therefore, the required number of learning data can be easily obtained. As a result, the labor and effort in obtaining the identification means can be more effectively reduced.

[0022] Means 3. The common direction image in the learning data and the inspection image data is set such that, while maintaining the commonality related to the squeegee movement direction, the long side or the short side extends along a predetermined direction. The solder inspection apparatus according to means 2, characterized in that.

[0023] According to the above-described means 3, the common-direction images in the learning data and the inspection image data are set such that while maintaining the commonality related to the moving direction of the squeegee, the long side or the short side extends along a predetermined direction. For example, the common-direction image constituting one piece of learning data is set such that the long side extends along the left-right direction and the moving direction of the squeegee is from right to left, while the common-direction image constituting another piece of learning data is set such that the long side extends along the left-right direction and, in order to maintain the commonality related to the moving direction of the squeegee, the moving direction of the squeegee is from upper right to lower left. By setting the learning data such that the long side or the short side of the common-direction image extends along a predetermined direction, the learning efficiency in the identification means can be further improved, and the labor and effort required to obtain the identification means can be further reduced. Also, by setting the inspection image data such that the long side or the short side of the common-direction image extends along a predetermined direction, the accuracy of the reconstructed image data can be further increased, and the inspection ability can be further improved.

[0024] Means 4. The solder inspection apparatus according to means 1, further comprising moving direction specifying means for specifying the moving direction of the squeegee when the cream solder contained in the image data acquired by the image data acquisition means is printed, based on the image data.

[0025] Note that, as the moving direction specifying means, for example, those that specify the moving direction of the squeegee based on the change in the height of the cream solder, or the positions of the volume center of gravity or the area center of gravity of the cream solder can be cited.

[0026] According to the above-described means 4, even when information regarding the moving direction of the squeegee is not input from an apparatus that prints the cream solder, an apparatus that controls the printing, or the like, it is possible to specify the moving direction of the squeegee.

[0027] Note that the technical matters related to the above-described means may be appropriately combined. Therefore, for example, the technical matters related to means 2 and the technical matters related to means 4 may be combined. [Brief explanation of the drawing]

[0028] [Figure 1] This is a magnified plan view of a portion of a printed circuit board. [Figure 2] This is a magnified cross-sectional view of a portion of a printed circuit board. [Figure 3] This is a block diagram showing the configuration of a printed circuit board manufacturing line. [Figure 4] This is a partially enlarged cross-sectional view showing how solder paste is printed by a squeegee moving in the forward direction. [Figure 5] This is a partially enlarged cross-sectional view showing how solder paste is printed using a squeegee moving in the opposite direction. [Figure 6] This is a partially enlarged cross-sectional view showing an example of solder paste printed by a squeegee moving in the forward direction. [Figure 7] This is a partially enlarged cross-sectional view showing an example of solder paste printed by a squeegee moving in the opposite direction. [Figure 8] This is a schematic diagram illustrating the configuration of a solder inspection device. [Figure 9] This is a block diagram showing the functional configuration of a solder inspection device. [Figure 10] This is a schematic diagram illustrating the structure of a neural network. [Figure 11] This is a flowchart showing the learning process of a neural network. [Figure 12] This is a flowchart showing the inspection process. [Figure 13] This is a schematic diagram showing the training source image data Ig1. [Figure 14] This is a schematic diagram showing the Ig2 image data used for training. [Figure 15] This is a schematic diagram showing the solder region image Hg1. [Figure 16] This is a schematic diagram showing the solder region image Hg2. [Figure 17] This is a schematic diagram showing the common-direction image Mg1. [Figure 18]This is a schematic diagram showing the direction-common image Mg2 obtained by rotation processing. [Figure 19] This is a schematic diagram showing the direction-common image Mg2 obtained by the inversion process. [Figure 20] This is a schematic diagram showing the image frame and training data G1. [Figure 21] This is a schematic diagram showing the image frame and training data G2. [Figure 22] This is a schematic diagram showing the original image data Ik1 used for inspection. [Figure 23] This is a schematic diagram showing the Ik2 value of the original image data used for inspection. [Figure 24] This is a schematic diagram showing the solder area image Hk1. [Figure 25] This is a schematic diagram showing the solder region image Hk2. [Figure 26] This is a schematic diagram showing the solder area image Hk3. [Figure 27] This is a schematic diagram showing the direction-common image Mk1. [Figure 28] This is a schematic diagram showing the direction-common image Mk2. [Figure 29] This is a schematic diagram showing the direction-common image Mk3 obtained by rotation processing. [Figure 30] This is a schematic diagram showing the direction-common image Mk3 obtained by the inversion process. [Figure 31] This is a schematic diagram showing the image frame and the image data K1 used for inspection. [Figure 32] This is a schematic diagram showing the image frame and the K2 image data for inspection. [Figure 33] This is a schematic diagram showing the image frame and the K3 image data for inspection. [Figure 34] This is a schematic diagram showing the reconstructed image data S1 output from the AI ​​model when the inspection image data K1 is input. [Figure 35] This is a schematic diagram showing the reconstructed image data S2 output from the AI ​​model when the inspection image data K2 is input. [Figure 36] This is a schematic diagram showing the reconstructed image data S3 output by the AI ​​model when the inspection image data K3 is input. [Figure 37] In another embodiment, this block diagram shows the functional configuration of a solder inspection device having two AI models. [Figure 38] In another embodiment, this is a schematic diagram showing the training source image data Ig4 and inspection source image data Ik4, where the longer side of the area occupied by the solder paste is oblique to the direction of movement of the squeegee. [Figure 39] This is a schematic diagram showing solder region images Hg4 and Hk4 in another embodiment. [Figure 40] This is a schematic diagram showing the common orientation images Mg4 and Mk4 in another embodiment. [Figure 41] This is a schematic diagram showing training data G4 and test image data K4 in another embodiment. [Modes for carrying out the invention]

[0029] The following describes one embodiment with reference to the drawings. First, the configuration of the printed circuit board will be described. Figure 1 is a partially enlarged plan view of a part of the printed circuit board, and Figure 2 is a partially enlarged cross-sectional view of a part of the printed circuit board.

[0030] As shown in Figures 1 and 2, the printed circuit board 1 consists of a flat base substrate 2 made of glass epoxy resin or the like, on which wiring patterns (not shown) made of copper foil and multiple rectangular lands 3 in plan view are formed. The portion of the base substrate 2 surface excluding the lands 3 is coated with a resist film 4.

[0031] Furthermore, solder paste 5, made by mixing solder granules with flux, is printed on land 3. In Figure 1, etc., for convenience, a scattered dot pattern is added to the area representing solder paste 5. Also, in Figure 1, etc., only a small portion of the multiple land 3 and solder paste 5 on the printed circuit board 1 are shown.

[0032] Next, the manufacturing line (manufacturing process) for producing the printed circuit board 1 will be explained with reference to Figure 3. As shown in Figure 3, the manufacturing line 10 is equipped with, in order from the upstream side (upper side of Figure 3), a solder printing machine 12, a solder inspection device 13, a component mounting machine 14, a reflow device 15, and a post-reflow inspection device 16.

[0033] The solder printing machine 12 performs a solder printing process to print solder paste 5 onto each land 3 of the printed circuit board 1. In the solder printing process, for example, as shown in Figures 4 and 5, the solder paste 5 is printed by screen printing.

[0034] In screen printing, first, with the lower surface of the screen mask 12a in contact with the printed circuit board 1, solder paste 5 is supplied to the upper surface of the screen mask 12a. In this embodiment, the screen mask 12a constitutes the "mask". The screen mask 12a has a plurality of openings 12a1 that correspond to each land 3 of the printed circuit board 1. Next, a predetermined squeegee 12b is moved while in contact with the upper surface of the screen mask 12a to fill the openings 12a1 with solder paste 5. After that, the printed circuit board 1 is separated from the lower surface of the screen mask 12a, and the solder paste 5 is printed on each land 3 of the printed circuit board 1.

[0035] Furthermore, the solder printing machine 12 sequentially prints solder paste 5 onto the inserted printed circuit boards 1. However, for each inserted printed circuit board 1, the direction of movement of the squeegee 12b when printing the solder paste 5 is alternately switched between the forward direction (for example, from right to left as shown in Figure 4) and the reverse direction (for example, from left to right as shown in Figure 5). For example, the solder printing machine 12 prints solder paste 5 onto odd-numbered printed circuit boards 1 by moving the squeegee 12b in the forward direction, and onto even-numbered printed circuit boards 1 by moving the squeegee 12b in the reverse direction. Therefore, among the multiple printed circuit boards 1 on which solder paste 5 has been printed by the solder printing machine 12, some may have the same direction of movement of the squeegee 12b when printing the solder paste 5, while others may have different directions of movement. Note that the solder printing machine 12 may also print solder paste 5 using a metal mask as a "mask".

[0036] Furthermore, the three-dimensional shape of the solder paste 5 may differ depending on the direction of movement of the squeegee 12b during printing. More specifically, the printed solder paste 5 tends to have a lower portion located upstream of the squeegee 12b's movement, while the portion located downstream tends to have a higher portion. Therefore, the manner in which the height changes on the upper surface of the solder paste 5 (the inclination of the upper surface) may differ depending on the direction of movement of the squeegee 12b during printing (see Figures 6 and 7). Figure 6 shows an example of solder paste 5 printed by moving the squeegee 12b in the forward direction, and Figure 7 shows an example of solder paste 5 printed by moving the squeegee 12b in the reverse direction.

[0037] The solder inspection device 13 inspects the shape of the solder paste 5 printed on the land 3. Details of the solder inspection device 13 will be described later.

[0038] The component mounting machine 14 mounts electronic components 25 (see Figures 1 and 2) on land 3 on which solder paste 5 is printed. The electronic component 25 has multiple electrodes (not shown), and each electrode is temporarily fixed to a predetermined amount of solder paste 5.

[0039] The reflow apparatus 15 heats and melts the solder paste 5 to perform a reflow process in which the land 3 and the electrodes of the electronic component 25 are soldered together.

[0040] The post-reflow inspection device 16 checks whether the solder joints were properly formed during the reflow process by, for example, using brightness image data to confirm whether there is any misalignment in the electronic components 25.

[0041] In addition, although not shown in the diagram, the manufacturing line 10 is equipped with conveyors for transporting printed circuit boards 1 between the aforementioned devices, such as between the solder printing machine 12 and the solder inspection device 13. Branching devices are also provided between the solder inspection device 13 and the component mounting machine 14, and downstream of the post-reflow inspection device 16. Printed circuit boards 1 that are judged as good by the solder inspection device 13 or the post-reflow inspection device 16 are guided downstream, while printed circuit boards 1 that are judged as defective are discharged to the defective product storage section by the branching devices.

[0042] Next, the configuration of the solder inspection device 13 will be explained in detail with reference to Figures 8 and 9. Figure 8 is a schematic diagram showing the configuration of the solder inspection device 13. Figure 9 is a block diagram showing the functional configuration of the solder inspection device 13.

[0043] The solder inspection device 13 includes a transport mechanism 31 that transports and positions the printed circuit board 1, an inspection unit 32 for obtaining image data of the printed circuit board 1, and a control device 33 (see Figure 9) that performs various controls, image processing, and calculation processing in the solder inspection device 13, including drive control of the transport mechanism 31 and the inspection unit 32.

[0044] The transport mechanism 31 comprises a pair of transport rails 31a arranged along the loading and unloading direction of the printed circuit board 1, and an endless conveyor belt 31b rotatably mounted on each transport rail 31a. Although not shown in the figures, the transport mechanism 31 is also provided with a drive means such as a motor for driving the conveyor belt 31b, and a chuck mechanism for positioning the printed circuit board 1 at a predetermined position. The transport mechanism 31 is driven and controlled by a control device 33 (a transport mechanism control unit 79, described later).

[0045] Under the above configuration, the printed circuit board 1, when brought into the solder inspection device 13, has both side edges in the width direction perpendicular to the loading / unloading direction inserted into the transport rails 31a and is placed on the conveyor belt 31b. Subsequently, the conveyor belt 31b starts moving, and the printed circuit board 1 is transported to a predetermined inspection position. When the printed circuit board 1 reaches the inspection position, the conveyor belt 31b stops and the chuck mechanism activates. This chuck mechanism pushes up the conveyor belt 31b, and both side edges of the printed circuit board 1 are clamped between the conveyor belt 31b and the upper edges of the transport rails 31a. This positions and fixes the printed circuit board 1 in the inspection position. When the inspection is completed, the fixing by the chuck mechanism is released, and the conveyor belt 31b starts moving again. As a result, the printed circuit board 1 is discharged from the solder inspection device 13. Of course, the configuration of the transport mechanism 31 is not limited to the above form, and other configurations may be adopted.

[0046] The inspection unit 32 is positioned above the transport rail 31a (the transport path for the printed circuit board 1). The inspection unit 32 is equipped with a first lighting device 32a, a second lighting device 32b, a third lighting device 32c, and a camera 32d.

[0047] Furthermore, the inspection unit 32 is also equipped with an X-axis movement mechanism 32e (see Figure 9) that enables movement in the X-axis direction (left-right direction in Figure 8), and a Y-axis movement mechanism 32f (see Figure 9) that enables movement in the Y-axis direction (front-back direction in Figure 8). These movement mechanisms 32e and 32f are driven and controlled by a control device 33 (movement mechanism control unit 76, which will be described later).

[0048] The first illumination device 32a and the second illumination device 32b, in order to perform three-dimensional measurement of the printed circuit board 1, each illuminate a predetermined area to be inspected on the printed circuit board 1 with predetermined light for three-dimensional measurement (patterned light having a striped light intensity distribution) from diagonally above.

[0049] Specifically, the first lighting device 32a includes a first light source 32a1 that emits predetermined light, and a first liquid crystal shutter 32a2 that forms a first grid that converts the light from the first light source 32a1 into first pattern light having a striped light intensity distribution, and is driven and controlled by a control device 33 (lighting control unit 72, which will be described later).

[0050] The second lighting device 32b includes a second light source 32b1 that emits predetermined light, and a second liquid crystal shutter 32b2 that forms a second grating that converts the light from the second light source 32b1 into a second pattern light having a striped light intensity distribution, and is driven and controlled by a control device 33 (lighting control unit 72, described later).

[0051] Under the above configuration, the light emitted from each light source 32a1 and 32b1 is guided to a focusing lens (not shown), where it is converted into parallel light, and then guided to a projection lens (not shown) via liquid crystal shutters 32a2 and 32b2, where it is projected onto the printed circuit board 1 as pattern light. In this embodiment, the switching control of the liquid crystal shutters 32a2 and 32b2 is performed so that the phase of each pattern light is shifted by a quarter pitch.

[0052] Furthermore, by using liquid crystal shutters 32a2 and 32b2 as the grating, it is possible to irradiate pattern light that is close to an ideal sine wave. This improves the measurement resolution of three-dimensional measurements. In addition, the phase shift of the pattern light can be controlled electrically, allowing for a more compact device.

[0053] The third illumination device 32c irradiates a predetermined area of ​​the printed circuit board 1 under inspection with a predetermined light (e.g., uniform light) for two-dimensional measurement of the printed circuit board 1. The third illumination device 32c is equipped with a ring light capable of emitting blue light, a ring light capable of emitting green light, and a ring light capable of emitting red light. Since the third illumination device 32c has a configuration similar to that of known technology, a detailed explanation thereof will be omitted.

[0054] Camera 32d captures a predetermined inspection area of ​​the printed circuit board 1 from directly above. Camera 32d has an image sensor such as a CCD (Charge Coupled Device) or CMOS (Complementary Metal Oxide Semiconductor) image sensor, and an optical system (lens unit, aperture, etc.) that forms an image of the printed circuit board 1 onto the image sensor, and its optical axis is arranged to align with the vertical direction (Z-axis direction). Of course, the image sensor is not limited to these, and other image sensors may be used.

[0055] Camera 32d is driven and controlled by control device 33 (camera control unit 73, described later). More specifically, control device 33 performs imaging processing by camera 32d in synchronization with the illumination processing by each illumination device 32a, 32b, and 32c. As a result, the light reflected from the printed circuit board 1 from any of the illumination devices 32a, 32b, and 32c is imaged by camera 32d. Consequently, image data of the area of ​​the printed circuit board 1 to be inspected, including the solder paste 5 printed on the land 3, is acquired. The "area to be inspected" of the printed circuit board 1 is one of several areas pre-set on the printed circuit board 1, with the size of the camera 32d's imaging field of view (imaging range) as one unit.

[0056] Furthermore, the camera 32d in this embodiment is a color camera. This allows for simultaneous imaging of each color of light illuminating the printed circuit board 1 from each color ring light of the third illumination device 32c.

[0057] Image data captured and generated by camera 32d is converted into a digital signal within camera 32d and then transferred in digital form to control device 33 (image acquisition unit 74, described later). The control device 33 then stores the transferred image data and performs various image processing and calculation processing based on the image data.

[0058] The control device 33 consists of a computer including a CPU (Central Processing Unit) that executes predetermined arithmetic processing, a ROM (Read Only Memory) that stores various programs and fixed value data, a RAM (Random Access Memory) that temporarily stores various data when executing various arithmetic processing, and peripheral circuits for these.

[0059] The control device 33 functions as various functional units, such as the main control unit 71, lighting control unit 72, camera control unit 73, image acquisition unit 74, data processing unit 75, movement mechanism control unit 76, learning unit 77, inspection unit 78, and transport mechanism control unit 79, as the CPU operates according to various programs.

[0060] However, the various functional units described above are realized through the cooperation of various hardware components such as the CPU, ROM, and RAM, and there is no need to clearly distinguish between functions realized in hardware and functions realized in software. Some or all of these functions may be realized by hardware circuits such as ICs.

[0061] Furthermore, the control device 33 includes an input unit 55 consisting of a keyboard, mouse, touch panel, etc., a display unit 56 equipped with a display screen consisting of a liquid crystal display, etc., a storage unit 57 capable of storing various data, programs, calculation results, inspection results, etc., and a communication unit 58 capable of sending and receiving various data with the outside.

[0062] Here, the various functional units that constitute the control device 33 will be described in detail.

[0063] The main control unit 71 is a functional unit that controls the entire solder inspection device 13 and is configured to send and receive various signals with other functional units such as the lighting control unit 72 and the camera control unit 73.

[0064] The lighting control unit 72 is a functional unit that drives and controls the lighting devices 32a, 32b, and 32c, and performs switching control of the illuminated light based on command signals from the main control unit 71.

[0065] The camera control unit 73 is a functional unit that drives and controls the camera 32d, and controls the imaging timing and other parameters based on command signals from the main control unit 71.

[0066] The image acquisition unit 74 is a functional unit for capturing image data acquired by the camera 32d.

[0067] The data processing unit 75 is a functional unit that performs predetermined image processing on image data acquired by the image acquisition unit 74 in response to instructions from other functional units, and performs two-dimensional measurement processing, three-dimensional measurement processing, etc. using the image data.

[0068] Furthermore, the data processing unit 75 performs three-dimensional measurement (height measurement) of the surface of the printed circuit board 1 using a known phase shift method or the like, based on the three-dimensional image data obtained by imaging the printed circuit board 1 with the camera 32d while pattern light is irradiated from the first illumination device 32a or the second illumination device 32b. This acquires height information at each position on the surface of the printed circuit board 1.

[0069] Furthermore, the data processing unit 75 acquires two-dimensional image data (hereinafter referred to as "image data with height information") in which height information is set for each pixel, based on the acquired height information and two-dimensional image data obtained by imaging the printed circuit board 1 with the camera 32d while uniform light is irradiated from the third illumination device 32c. The height information in the image data with height information can also be indicated by differences in brightness or color, for example. In this embodiment, the camera 32d and the data processing unit 75 constitute the "image data acquisition means".

[0070] Furthermore, the data processing unit 75 identifies the direction of movement of the squeegee 12b when printing the solder paste 5 included in the image data, based on the image data with height information. More specifically, the data processing unit 75 identifies the area occupied by the solder paste 5 in the image data with height information, and calculates information (e.g., inclination) indicating the manner of height change in the identified area (i.e., the inclination of the upper surface of the solder paste 5). Then, based on the calculated information, the data processing unit 75 identifies the direction of movement of the squeegee 12b as either the forward direction (e.g., from right to left) or the reverse direction (e.g., from left to right). Note that the solder paste 5 targeted for information calculation may be all of the solder paste 5 included in the image data with height information, or it may be only some of the solder paste 5. In this embodiment, the data processing unit 75 constitutes the "movement direction identification means".

[0071] The movement mechanism control unit 76 is a functional unit that drives and controls the X-axis movement mechanism 32e and the Y-axis movement mechanism 32f, and controls the position of the inspection unit 32 based on command signals from the main control unit 71. By driving and controlling the X-axis movement mechanism 32e and the Y-axis movement mechanism 32f, the movement mechanism control unit 76 can move the inspection unit 32 to a position above any area to be inspected on the printed circuit board 1, which is positioned and fixed at the inspection position. Then, as the inspection unit 32 is sequentially moved to multiple areas to be inspected set on the printed circuit board 1, the inspection of the area to be inspected is performed, thereby performing an inspection of the entire printed circuit board 1.

[0072] The learning unit 77 is a functional unit that uses training data to train the deep neural network 90 (hereinafter simply referred to as "neural network 90"; see Figure 10) and constructs an AI (Artificial Intelligence) model 101 as a "discrimination tool".

[0073] As will be described later, the AI ​​model 101 in this embodiment is a generative model constructed by deep learning a neural network 90 using only image data related to good solder paste 5 as training data, and has the structure of a so-called autoencoder.

[0074] Here, the structure of the neural network 90 will be explained with reference to Figure 10. Figure 10 is a schematic diagram conceptually showing the structure of the neural network 90. ​​As shown in Figure 10, the neural network 90 has the structure of a convolutional auto-encoder (CAE), comprising an encoder unit 91 as an "encoding unit" that extracts feature quantities (latent variables) TA from the input image data GA, and a decoder unit 92 as a "decoding unit" that reconstructs image data GB from the feature quantities TA.

[0075] The structure of the convolutional autoencoder is well known, so a detailed explanation will be omitted. The encoder unit 91 has multiple convolutional layers 93, and in each convolutional layer 93, the result of a convolution operation using multiple filters (kernels) 94 on the input data is output as input data for the next layer. Similarly, the decoder unit 92 has multiple deconvolutional layers 95, and in each deconvolutional layer 95, the result of a deconvolution operation using multiple filters (kernels) 96 on the input data is output as input data for the next layer. Then, in the learning process described later, the weights (parameters) of each filter 94, 96 are updated.

[0076] The inspection unit 78 is a functional unit that inspects for abnormalities related to the solder paste 5. In this embodiment, the inspection unit 78 inspects whether the solder paste 5 is properly printed in terms of size, shape, etc.

[0077] The transport mechanism control unit 79 is a functional unit that drives and controls the transport mechanism 31, and controls the position of the printed circuit board 1 based on command signals from the main control unit 71.

[0078] The memory unit 57 is composed of an HDD (Hard Disk Drive) or an SSD (Solid State Drive), and has a predetermined memory area for storing, for example, the AI ​​model 101 (neural network 90 and the learning information acquired through its learning).

[0079] The communication unit 58 is equipped with a communication interface conforming to communication standards such as wired LAN (Local Area Network) or wireless LAN, and is configured to send and receive various data to and from the outside. For example, the results of inspections performed by the inspection unit 78 are output to the outside via the communication unit 58, and the results of inspections performed by the post-reflow inspection device 16 are input via the communication unit 58.

[0080] Next, the learning process of the neural network 90 performed by the solder inspection device 13 will be explained with reference to the flowchart in Figure 11.

[0081] When the learning process starts based on the execution of a predetermined learning program, the main control unit 71 first performs pre-processing in step S101 for training the neural network 90.

[0082] In this preprocessing step, inspection information for numerous printed circuit boards 1 stored in the post-reflow inspection device 16 is first acquired via the communication unit 58. Subsequently, based on this inspection information, learning source image data Ig1 and Ig2, which are image data related to good solder paste 5 that passed the post-reflow inspection, are acquired from the storage unit 57 (see, for example, Figures 13 and 14). Figures 13 and 14 show only a portion of the learning source image data Ig1 and Ig2; the actual learning source image data Ig1 and Ig2 contain many more solder paste 5 and lands 3. Also, in Figure 13, etc., for ease of explanation, the approximate height differences in the solder paste 5 are shown using the shading of a scattered dot pattern. Furthermore, in Figure 13, etc., the direction of movement of the squeegee 12b when printing the solder paste 5 is indicated by a white arrow.

[0083] The training source image data Ig1 and Ig2 relate to the printed circuit board 1 after the solder paste 5 has been printed and before the electronic components 25 have been mounted, and are used to obtain the training data G1 and G2 described later for training the neural network 90. ​​The training source image data Ig1 and Ig2 are each image data with height information. The training source image data Ig1 relates to the printed circuit board 1 on which the solder paste 5 has been printed by moving the squeegee 12b in the forward direction, and the training source image data Ig2 relates to the printed circuit board 1 on which the solder paste 5 has been printed by moving the squeegee 12b in the reverse direction.

[0084] Next, the areas occupied by the solder paste 5 in the acquired training image data Ig1 and Ig2 are identified. The areas occupied by the solder paste 5 are identified, for example, based on two-dimensional image data obtained by imaging the printed circuit board 1 with the camera 32d while uniform light is irradiated from the third illumination device 32c, using luminance, hue, saturation, etc.

[0085] Next, images of the connected components (clumps) in the area occupied by the identified solder paste 5 are extracted as solder region images Hg1 and Hg2 (see, for example, Figures 15 and 16). Figures 15, etc., show some examples of solder region images Hg1 and Hg2. The numerical values ​​in Figures 15, etc., represent examples of the height in each part of the solder paste 5.

[0086] The solder region images Hg1 and Hg2 correspond to the land 3 of 1. In this embodiment, the connected components (clumps) of the area occupied by the solder paste 5 that are located on the land 3 of 1 according to the design are extracted as the solder region images Hg1 and Hg2 of 1. Alternatively, the connected components (clumps) of 1 may be extracted as the solder region images Hg1 and Hg2 of 1 without considering the position of the land 3.

[0087] Next, common direction images Mg1 and Mg2 (see Figures 17 and 18) are obtained from the solder region images Hg1 and Hg2. The common direction images Mg1 and Mg2 are images in which the solder region images Hg1 and Hg2 are set so that the movement direction of the squeegee 12b related to the solder region images Hg1 and Hg2 is a predetermined direction (forward direction in this embodiment). The movement direction of the squeegee 12b related to the solder region images Hg1 and Hg2 is obtained by the data processing unit 75.

[0088] In this embodiment, the solder region image Hg1 is obtained when the squeegee 12b is moving in a predetermined direction (forward direction), and is therefore used as is as the common direction image Mg1. On the other hand, the solder region image Hg2 is obtained when the squeegee is moving in the opposite direction to the predetermined direction, so the common direction image Mg2 is obtained by rotating the solder region image Hg2 by 180°. Alternatively, the common direction image Mg2 may be obtained by performing an inversion process instead of rotation (see Figure 19).

[0089] Then, by pasting the direction-common images Mg1 and Mg2 onto the image frame W1, we obtain training data G1 and G2 (see Figures 20 and 21) in which the direction-common images Mg1 and Mg2 are placed on the image frame W1. Since the direction-common images Mg1 and Mg2 have the same direction of movement for the squeegee 12b, the resulting training data G1 and G2 will have the same direction of movement for the squeegee 12b.

[0090] Furthermore, the image frame W1 is a rectangular image with a height (width in the vertical direction of the paper, such as in Figure 20) of m (pixels) and a width (width in the horizontal direction of the paper, such as in Figure 20) of n (pixels), with the height information of each pixel set to 0. The size (width and height) of the image frame W1 is set to be larger than the sizes of the common direction images Mg1 and Mg2, based on design data, etc. Note that m and n are natural numbers, and in this embodiment, m=n is satisfied. In addition, by adjusting the pasting position, each learning data G1 and G2 is set to coincide with the center or centroid of the common direction images Mg1 and Mg2 and the center of the image frame W1.

[0091] Then, by repeatedly performing the above processes of extracting solder region images Hg1 and Hg2, acquiring direction-common images Mg1 and Mg2, and pasting direction-common images Mg1 and Mg2 onto image frame W1, multiple training data G1 and G2 are obtained from each training source image data Ig1 and Ig2. Furthermore, by using the training source image data Ig1 and Ig2 related to multiple printed circuit boards 1, the required number of training data G1 and G2 are ultimately obtained.

[0092] In step S101, once the necessary number of training data G1 and G2 have been acquired, in the following step S102, the learning unit 77 prepares an untrained neural network 90 based on a command from the main control unit 71. For example, it reads a neural network 90 that has been previously stored in the memory unit 57 or the like. Alternatively, it constructs a neural network 90 based on network configuration information (for example, the number of layers in the neural network and the number of nodes in each layer) stored in the memory unit 57 or the like.

[0093] In step S103, reconstructed image data is acquired. Specifically, based on a command from the main control unit 71, the learning unit 77 provides the learning data G1 and G2 acquired in step S102 as input data to the input layer of the neural network 90, thereby acquiring the reconstructed image data output from the output layer of the neural network 90.

[0094] In the following step S104, the learning unit 77 compares the input learning data G1 and G2 with the reconstructed image data output by the neural network 90 and determines whether the error is sufficiently small (whether it is below a predetermined threshold).

[0095] Here, if the error is sufficiently small, in step S106, the learning unit 77 determines whether the learning termination conditions are met. For example, if a certain number of consecutive affirmative judgments are made in step S104 without going through the process of step S105 described later, or if learning using all of the prepared learning data G1 and G2 is repeated a certain number of times, it is determined that the termination conditions are met. If the termination conditions are met, the neural network 90 and its learning information (updated parameters, etc., described later) are stored in the storage unit 57 as the AI ​​model 101, and this learning process is terminated. Therefore, the AI ​​model 101 is generated by learning only image data related to good solder paste 5 in which the movement direction of the squeegee 12b is common, as the learning data G1 and G2.

[0096] On the other hand, if the termination condition is not met in step S106, the process returns to step S102 and the neural network 90 is trained again.

[0097] Furthermore, if the error is not sufficiently small in step S104, the network update process (training of the neural network 90) is performed in step S105, and then the process returns to step S103 and the above series of processes is repeated.

[0098] Specifically, in the network update process of step S105, known learning algorithms such as backpropagation are used to update the weights (parameters) of each filter 94 and 96 in the neural network 90 to more appropriate values ​​so that the loss function representing the difference between the training data G1 and G2 and the reconstructed image data is minimized. For example, BCE (Binary Cross-entropy) can be used as the loss function.

[0099] By repeatedly performing steps S103 to S105, the neural network 90 minimizes the error between the training data G1 and G2 and the reconstructed image data, resulting in the output of more accurate reconstructed image data.

[0100] The resulting AI model 101 will generate reconstructed image data that closely matches the image data of a good solder paste 5 when that image data is input. Furthermore, when the AI ​​model 101 receives image data of a defective solder paste 5 in terms of shape, size, etc., it will generate reconstructed image data that closely matches the image data of the solder paste 5 after the shape and size have been corrected. In other words, when the solder paste 5 is defective, the reconstructed image data of the solder paste 5 will be a virtual image data of the solder paste 5 assuming that there are no defects.

[0101] Next, the inspection process performed by the solder inspection device 13 will be explained with reference to the flowchart in Figure 12. This inspection process is performed for each area to be inspected on the printed circuit board 1.

[0102] Once the printed circuit board 1 is brought into the solder inspection device 13 and positioned at a predetermined inspection location, the inspection process is started based on the execution of a predetermined inspection program.

[0103] When the inspection process begins, the first step, S301, is the image data acquisition process. In the image data acquisition process, the raw inspection image data Ik1 and Ik2 (see, for example, Figures 22 and 23) related to the printed circuit board 1 to be inspected are acquired. The raw inspection image data Ik1 and Ik2 are image data used to obtain the inspection image data K1, K2, and K3 (hereinafter sometimes simply referred to as "inspection image data K1 to K3"), which will be described later.

[0104] In this embodiment, examples of printed circuit boards 1 to be inspected include those with an abnormality (missing part) in the two-dimensional shape of the solder paste 5 (see Figure 22) and those with an abnormality (insufficient height) in the three-dimensional shape of the solder paste 5 (see Figure 23).

[0105] The image data acquisition process first involves acquiring three-dimensional image data and then two-dimensional image data.

[0106] First, the process for acquiring three-dimensional image data will be explained. In this process, the phase of the first pattern light emitted from the first illumination device 32a is changed, and imaging is performed four times under the first pattern light with different phases. Then, the phase of the second pattern light emitted from the second illumination device 32b is changed, and imaging is performed four times under the second pattern light with different phases, thereby acquiring a total of eight different three-dimensional image data. This will be explained in detail below.

[0107] As described above, once the printed circuit board 1, which has been brought into the solder inspection device 13, is positioned and fixed in a predetermined inspection position, the movement mechanism control unit 76, based on a command from the main control unit 71, first drives and controls the X-axis movement mechanism 32e and the Y-axis movement mechanism 32f to move the inspection unit 32, and adjusts the imaging field of view (imaging range) of the camera 32d to the predetermined area to be inspected on the printed circuit board 1.

[0108] In addition, the lighting control unit 72 switches the liquid crystal shutters 32a2 and 32b2 of both lighting devices 32a and 32b, and sets the positions of the first and second grids formed on both liquid crystal shutters 32a2 and 32b2 to predetermined reference positions.

[0109] Once the switching settings for the first and second grids are complete, the lighting control unit 72 causes the first light source 32a1 of the first lighting device 32a to emit light and irradiate with the first pattern light, while the camera control unit 73 drives and controls the camera 32d to perform the first imaging process under the first pattern light. The image data generated by the imaging process is taken into the image acquisition unit 74 as needed (the same applies hereafter). As a result, three-dimensional image data of the area under inspection, including multiple lands 3 and multiple solder pastes 5, is acquired.

[0110] Subsequently, the lighting control unit 72, simultaneously with the completion of the first imaging process under the first pattern light, turns off the first light source 32a1 of the first lighting device 32a and performs a switching process for the first liquid crystal shutter 32a2. Specifically, it switches the position of the first grid formed on the first liquid crystal shutter 32a2 from the reference position to a second position where the phase of the first pattern light is shifted by a quarter pitch (90°).

[0111] Once the switching settings for the first grid are complete, the lighting control unit 72 causes the light source 32a1 of the first lighting device 32a to emit light, illuminating with the first pattern light, and the camera control unit 73 drives and controls the camera 32d to perform a second imaging process under the first pattern light. Thereafter, the same process is repeated to acquire four different three-dimensional image data under the first pattern light with a phase difference of 90°.

[0112] Next, the lighting control unit 72 causes the second light source 32b1 of the second lighting device 32b to emit light and irradiate with the second pattern light, while the camera control unit 73 drives and controls the camera 32d to perform the first imaging process under the second pattern light.

[0113] Subsequently, the lighting control unit 72, simultaneously with the completion of the first imaging process under the second pattern light, turns off the second light source 32b1 of the second illumination device 32b and performs a switching process for the second liquid crystal shutter 32b2. Specifically, it switches the position of the second grid formed on the second liquid crystal shutter 32b2 from the reference position to a second position where the phase of the second pattern light is shifted by a quarter pitch (90°).

[0114] Once the switching settings for the second grid are complete, the lighting control unit 72 causes the light source 32b1 of the second lighting device 32b to emit light, illuminating with the second pattern light, and the camera control unit 73 drives and controls the camera 32d to perform a second imaging process under the second pattern light. Thereafter, the same process is repeated to acquire four types of three-dimensional image data under the second pattern light with a phase difference of 90°.

[0115] Next, the process of acquiring two-dimensional image data will be described. In this process, based on a command from the main control unit 71, the illumination control unit 72 causes the third illumination device 32c to emit light, irradiating a predetermined area to be inspected with uniform light, while the camera control unit 73 drives and controls the camera 32d to perform imaging under the uniform light. As a result, a predetermined area to be inspected on the printed circuit board 1 is imaged, and two-dimensional image data relating to the area to be inspected is acquired.

[0116] Next, based on the acquired three-dimensional image data, three-dimensional measurement (height measurement) of the surface of the printed circuit board 1 is performed using a phase shift method or the like. Furthermore, based on the height information acquired by the three-dimensional measurement and the two-dimensional image data, inspection source image data Ik1 and Ik2, in which height information is set for each pixel, are acquired. Inspection source image data Ik1 and Ik2 are both image data with height information. The acquired inspection source image data Ik1 and Ik2 are stored in the storage unit 57.

[0117] Next, in step S302, the inspection image data acquisition process is executed. In the inspection image data acquisition process, inspection image data K1 to K3 (see Figures 31 to 33) are acquired based on the original inspection image data Ik1 and Ik2 obtained in the image data acquisition process. The method for acquiring inspection image data K1 to K3 is the same as the method for acquiring the training data G1 and G2 described above.

[0118] Specifically, first, the area occupied by solder paste 5 in the acquired raw inspection image data Ik1 and Ik2 is identified. Next, solder area images Hk1, Hk2, and Hk3 (see, for example, Figures 24-26) of the identified area occupied by solder paste 5 are extracted. Figures 24, etc., show some of the solder area images Hk1, Hk2, and Hk3 (hereinafter sometimes simply referred to as "solder area images Hk1-Hk3") as an example.

[0119] The solder region images Hk1 to Hk3 of 1 correspond to the land 3 of 1. In this embodiment, the entire image of the connected components (block portions) in the identified region that overlap at least partially with the land 3 of 1 in the design data or manufacturing data is extracted as the solder region images Hk1 to Hk3 of 1. Therefore, if there are two or more connected components in the land 3 of 1 in the design data or manufacturing data, the solder region images Hk1 to Hk3 of 1 are composed of two or more connected components. Also, if a part of a connected component extends beyond the land 3 in the design data or manufacturing data, the solder region images Hk1 to Hk3 of 1 are composed of the entire connected component including the part that extends beyond the land 3. Note that the connected components of 1 may simply be extracted as the solder region images Hk1 to Hk3 of 1 without using design data, etc.

[0120] Next, common direction images Mk1, Mk2, and Mk3 (see Figures 27-29) are obtained from the identified solder region images Hk1-Hk3. The common direction images Mk1, Mk2, and Mk3 (hereinafter sometimes simply referred to as "common direction images Mk1-Mk3") are images in which the solder region images Hk1-Hk3 are set such that the direction of movement of the squeegee 12b related to the solder region images Hk1-Hk3 is the same direction (forward direction in this embodiment) as the direction of movement of the squeegee 12b in the training data G1 and G2. The direction of movement of the squeegee 12b related to the solder region images Hk1-Hk3 is obtained by the data processing unit 75.

[0121] In this embodiment, the solder region images Hk1 and Hk2 are used as the common direction images Mk1 and Mk2 because the direction of movement of the squeegee 12b is the same as the direction of movement of the squeegee 12b in the training data G1 and G2. On the other hand, the direction of movement of the squeegee 12b in the solder region image Hk3 is the opposite direction to the direction of movement of the squeegee 12b in the training data G1 and G2. Therefore, the common direction image Mk3 is obtained by rotating the solder region image Hk3 by 180°. Alternatively, the common direction image Mk3 may be obtained by performing a flipping process instead of a rotation process (see Figure 30).

[0122] Next, by attaching the direction-common images Mk1 to Mk3 to the image frame W1, inspection image data K1 to K3 are obtained, each containing the direction-common images Mk1 to Mk3 on the image frame W1 (see Figures 31 to 33). Since the image frame W1 of the inspection image data K1 to K3 is the same size as the image frame W1 of the training data G1 and G2, the size (width and height) of the inspection image data K1 to K3 is the same as the size of the training data G1 and G2. Also, since the direction-common images Mk1 to Mk3 have the same movement direction of the squeegee 12b, the resulting inspection image data K1 to K3 will have the same movement direction of the squeegee 12b.

[0123] Then, by repeatedly performing the above processes of extracting solder area images Hk1 to Hk3, acquiring direction common images Mk1 to Mk3, and pasting direction common images Mk1 to Mk3 onto the image frame W1, multiple inspection image data K1 to K3 are obtained from each of the original inspection image data Ik1 and Ik2. In this embodiment, the inspection unit 78, which acquires inspection image data K1 to K3 in cooperation with the data processing unit 75, constitutes the "inspection image data acquisition means".

[0124] In the subsequent step S303, the reconstruction image data acquisition process is performed. Specifically, based on a command from the main control unit 71, the inspection unit 78 inputs the inspection image data K1 to K3 acquired in step S302 into the input layer of the AI ​​model 101. Accordingly, the inspection unit 78 inputs the inspection image data K1 to K3 into the input layer of the AI ​​model 101 such that the direction of movement of the squeegee 12b in the learned training data G1 and G2 is the same as the direction of movement of the squeegee 12b related to the solder paste 5 to be inspected, which is included in the inspection image data K1 to K3.

[0125] Then, the image data reconstructed by the AI ​​model 101 and output from the output layer is acquired as reconstructed image data S1, S2, and S3 (see Figures 34-36). The acquired reconstructed image data S1, S2, and S3 (hereinafter sometimes simply referred to as "reconstructed image data S1-S3") are stored in association with the inspection image data K1-K3 from which the reconstructed image data S1-S3 were derived.

[0126] Here, if the AI ​​model 101 receives, for example, inspection image data K1 (see Figure 31) relating to solder paste 5 with a two-dimensional shape abnormality, it will output, as reconstructed image data S1, image data relating to a good solder paste 5 with the two-dimensional shape abnormality (defect) corrected (see, for example, Figure 34). Also, if the AI ​​model 101 receives, for example, inspection image data K3 (see Figure 33) relating to solder paste 5 with a three-dimensional shape abnormality, it will output, as reconstructed image data S3, image data relating to a good solder paste 5 with the three-dimensional shape abnormality (insufficient height) corrected (see, for example, Figure 36).

[0127] On the other hand, if the AI ​​model 101 receives, for example, inspection image data K2 relating to good solder paste 5, it outputs reconstructed image data S2 relating to good solder paste 5 that is almost identical to the inspection image data K2. The size (width and height) of the reconstructed image data S1 to S3 is the same as the size of the original inspection image data K1 to K3. In this embodiment, the inspection unit 78 that acquires the reconstructed image data S1 to S3 constitutes the "reconstructed image data acquisition means".

[0128] In step S304, a pass / fail judgment process is performed based on the acquired reconstructed image data S1 to S3. In the pass / fail judgment process, based on a command from the main control unit 71, the inspection unit 78 compares the entirety of the inspection image data K1 to K3 acquired in step S302 with the entirety of the reconstructed image data S1 to S3 acquired in step S303 using the inspection image data K1 to K3, and calculates the difference between the two image data sets K1 to K3 and S1 to S3. For example, by comparing dots (pixels) at the same coordinates in both image data sets K1 to K3 and S1 to S3, the area (number of dots) of clusters of dots where the height difference is greater than or equal to a predetermined value is calculated. Note that the difference between the inspection image data K1 and K3, in which the solder paste 5 has an abnormality, and the reconstructed image data S1 and S3, composed of the inspection image data K1 and K3, will be relatively large. On the other hand, the difference between the inspection image data K2, in which the solder paste 5 does not have an abnormality, and the reconstructed image data S2, composed of the inspection image data K2, will be relatively small. In this embodiment, the inspection unit 78 that compares inspection image data K1 to K3 and reconstructed image data S1 to S3 constitutes the "comparison means".

[0129] Next, the inspection unit 78 determines whether the calculated difference is smaller than a predetermined threshold. If the calculated difference is smaller than the predetermined threshold, the inspection unit 78 determines it to be a "good product," while if the difference is larger than the predetermined threshold, it determines it to be a "defective product."

[0130] Furthermore, the inspection unit 78 performs the above determination on all inspection image data K1 to K3 related to the area to be inspected on the printed circuit board 1. If all inspection image data K1 to K3 are determined to be "good," the inspection unit 78 determines that the area to be inspected is "good" and stores this result in the storage unit 57. On the other hand, if, as a result of performing the above determination on all inspection image data K1 to K3 related to the area to be inspected, at least one inspection image data K1 to K3 is determined to be "defective," the inspection unit 78 determines that the area to be inspected is "defective" and stores this result in the storage unit 57.

[0131] Then, if the solder inspection device 13 determines that all areas to be inspected on the printed circuit board 1 are "good" as a result of the inspection process, it determines that the printed circuit board 1 is free of defects in the solder paste 5 (pass judgment) and stores this result in the storage unit 57.

[0132] On the other hand, if the solder inspection device 13 finds even one area under inspection that is determined to be a "defective product," it determines that the printed circuit board 1 has an abnormality in the solder paste 5 (failure judgment), stores this result in the storage unit 57, and notifies the outside of this fact via the display unit 56, communication unit 58, etc.

[0133] As detailed above, according to this embodiment, the inspection image data K1 to K3 are formed by providing the direction-common images Mk1 to Mk3 in the image frame W1. Therefore, the size (width and height) of the inspection image data K1 to K3 will match the size of the image frame W1 and will remain constant without fluctuating in detail depending on the size of the land 3 (solder paste 5). This eliminates the need to prepare multiple different AI models for each land 3 size (size of the inspection image data).

[0134] Furthermore, the training data G1 and G2 are obtained by setting the direction of movement of the squeegee 12b to a predetermined direction and placing the common direction images Mg1 and Mg2 in the image frame W1. Similarly, the inspection image data K1 to K3 are obtained by setting the direction of movement of the squeegee 12b to a direction common to that direction in the training data G1 and G2 and placing the common direction images Mk1 to Mk3 in the image frame W1. Consequently, the direction of movement of the squeegee 12b in the inspection image data K1 to K3 is consistent, and the AI ​​model 101 is adjusted (tuned) to match the direction of movement of the squeegee 12b. Therefore, the accuracy of the reconstructed image data S1 to S3, which is reconstructed when the inspection image data K1 to K3 is input to the AI ​​model 101, can be improved, and superior inspection capabilities can be obtained more reliably. In addition, there is no need to prepare multiple different AI models for each direction of movement of the squeegee 12b in the inspection image data. As a result, combined with the fact that it is not necessary to prepare multiple different AI models for each size of land 3 as described above, the effort and time required to obtain AI model 101 can be reduced. In addition, AI model 101 can be used in common regardless of the size of land 3 or the direction of movement of squeegee 12b.

[0135] In addition, the image frame W1 of the training data G1 and G2 and the image frame W1 of the inspection image data K1 to K3 are the same size, and the sizes of the training data G1 and G2 and the inspection image data K1 to K3 are identical. Therefore, when the inspection image data K1 to K3 is input to the AI ​​model 101, it is possible to output appropriate reconstructed image data S1 to S3 corresponding to the inspection image data K1 to K3 with greater reliability. This makes it possible to inspect the solder paste 5 with greater accuracy.

[0136] Furthermore, from multiple solder region images Hg1 and Hg2 with different movement directions of the squeegee 12b, training data G1 and G2 with a common movement direction of the squeegee 12b can be obtained for training AI model 101. Therefore, the required number of training data G1 and G2 can be easily obtained. As a result, the effort and time required to obtain AI model 101 can be reduced even more effectively.

[0137] Furthermore, the data processing unit 75 determines the direction of movement of the squeegee 12b when printing the solder paste 5 contained in the image data, based on the image data with height information (training source image data Ig1, Ig2 and inspection source image data Ik1, Ik2). Therefore, even if information regarding the direction of movement of the squeegee 12b is not input to the solder inspection device 13 from the solder printing machine 12 or the like, the direction of movement of the squeegee 12b can be determined.

[0138] Furthermore, the embodiment is not limited to the description above, and may be implemented as follows, for example. Of course, other applications and modifications not exemplified below are also possible.

[0139] (a) In the above embodiment, an AI model 101 is generated by training it with training data G1 and G2 in which the movement direction of the squeegee 12b is aligned, and inspection image data K1 to K3 in which the movement direction of the squeegee 12b is similarly aligned is input to the AI ​​model 101.

[0140] Alternatively, as shown in Figure 37, a first AI model 103 may be generated by training only on training data where the squeegee 12b moves in the forward direction, and a second AI model 104 may be generated by training only on training data where the squeegee 12b moves in the reverse direction. Then, inspection image data where the squeegee 12b moves in the forward direction may be input to the first AI model 103, and inspection image data where the squeegee 12b moves in the reverse direction may be input to the second AI model 104. In other words, an AI model 103 corresponding to the forward direction and an AI model 104 corresponding to the reverse direction may be generated separately, and one of the AI ​​models 103 or 104 may be selected according to the direction of movement of the squeegee 12b in the inspection image data, and the inspection image data may be input to the selected AI model 103 or 104.

[0141] In this case as well, the AI ​​models 103 and 104, which receive the inspection image data, are adjusted (tuned) to match the direction of movement of the squeegee 12b in that inspection image data. Therefore, the accuracy of the reconstructed image data reconstructed when the inspection image data is input to the AI ​​models 103 and 104 can be improved, and superior inspection capabilities can be obtained more reliably.

[0142] (b) In the above embodiment, the solder inspection device 13 is configured to determine the direction of movement of the squeegee 12b when the solder paste 5 included in the image data is printed, based on image data with height information (learning source image data Ig1, Ig2 and inspection source image data Ik1, Ik2). That is, the solder inspection device 13 is configured to determine the direction of movement of the squeegee 12b related to the solder paste 5 from the state of the printed solder paste 5.

[0143] In contrast, the solder inspection device 13 may identify the direction of movement of the squeegee 12b for each printed circuit board 1 based on information regarding the direction of movement of the squeegee 12b for each printed circuit board 1 input from the solder printing machine 12. In this case, it is not necessary to perform a process to determine the method of movement of the squeegee 12b from the state of the solder paste 5, thus speeding up the inspection process.

[0144] (c) In the above embodiment, the area occupied by the solder paste 5 in the training source image data Ig1, Ig2 and the inspection source image data Ik1, Ik2 is defined as having a long side parallel to the direction of movement of the squeegee 12b. Therefore, the direction-common images Mg1, Mg2, Mk1~Mk3 in the training data G1, G2 and inspection image data K1~K3 have their long sides extending in a predetermined direction (left-right direction) without any special image processing.

[0145] In contrast, as shown in Figure 38, for example, if the long or short side of the area occupied by the solder paste 5 is oblique to the direction of movement of the squeegee 12b in the training data Ig4 or the inspection data Ik4, the direction-common images Mg4 and Mk4 in the training data G4 and inspection data K4 may be set so that the long or short side extends in a predetermined direction (for example, left or right) while maintaining commonality with respect to the direction of movement of the squeegee 12b (see Figure 41).

[0146] To obtain the training data G4 and inspection image data K4, first, as in the above embodiment, solder region images Hg4 and Hk4 are extracted from the training source image data Ig4 and the inspection source image data Ig4 (see Figure 39). Then, while maintaining the commonality of the direction of movement of the squeegee 12b (that is, while maintaining the state in which one directional component of the direction of movement of the squeegee 12b (the direction from right to left) is the same as the direction of movement of the squeegee 12b in the other direction common images Mg1, Mg2, Mk1~Mk3), the solder region images Hg4 and Hk4 are rotated. As a result, direction common images Mg4 and Mk4 are obtained in which the long side or short side extends in a predetermined direction while maintaining the commonality of the direction of movement of the squeegee 12b (see Figure 40). Then, as in the above embodiment, the training data G4 and inspection image data K4 can be obtained by placing the direction common images Mg4 and Mk4 in the image frame W1.

[0147] By setting the training data G4 such that the long or short side of the directional common image Mg4 extends along a predetermined direction, the learning efficiency of the AI ​​model 101 can be further improved, and the effort and time required to obtain the AI ​​model 101 can be further reduced. In addition, by setting the inspection image data K4 such that the long or short side of the directional common image Mk4 extends along a predetermined direction, the accuracy of the reconstructed image data can be further improved, and the inspection capability can be further enhanced.

[0148] (d) In the above embodiment, the direction of movement of the squeegee 12b is determined based on information indicating the change in height in the area occupied by the solder paste 5 (e.g., inclination). Alternatively, the direction of movement of the squeegee 12b may be determined based on the area centroid of the area occupied by the solder paste 5 when viewed from the side, or the centroid position of the solder paste 5 obtained from three-dimensional image data. For example, when the solder paste 5 is viewed from the side, if the area centroid is shifted to the right of the volume centroid, the direction of movement of the squeegee 12b can be determined to be from right to left, and if the area centroid is shifted to the left of the volume centroid, the direction of movement of the squeegee can be determined to be from left to right.

[0149] (e) In the above embodiment, when training the neural network 90, training data G1 and G2 are obtained using training source image data Ig1 and Ig2 relating to the printed circuit board 1 that has passed the post-reflow inspection. Alternatively, training data G1 and G2 may be obtained using training source image data relating to good quality solder paste 5 that has been visually selected by an operator after printing the solder paste 5.

[0150] Alternatively, the learning unit 77 may acquire learning data G1 and G2 using image data of a virtually generated good solder paste 5.

[0151] (f) The configuration of the AI ​​model 101 (neural network 90) and its learning method are not limited to the above embodiment. For example, when performing the learning process of the neural network 90 or the reconstruction image data acquisition process, the configuration may include normalization or other processing on various data as needed. Also, the structure of the neural network 90 is not limited to that shown in Figure 10, and for example, a pooling layer may be provided after the convolutional layer 93. Of course, the number of layers in the neural network 90, the number of nodes in each layer, and the connection structure of each node may also be different.

[0152] Furthermore, in the above embodiment, the AI ​​model 101 (neural network 90) is a generative model having the structure of a convolutional autoencoder (CAE), but it is not limited to this, and may be a generative model having the structure of a different type of autoencoder, such as a variational autoencoder (VAE).

[0153] Furthermore, in the above embodiment, the neural network 90 is trained using the backpropagation method, but the system is not limited to this, and various other learning algorithms may be used for training.

[0154] In addition, the neural network 90 may be composed of a dedicated AI processing circuit, such as a so-called AI chip. In that case, only learning information such as parameters may be stored in the memory unit 57, and the dedicated AI processing circuit may read this information and set it in the neural network 90 to construct the AI ​​model 101.

[0155] In addition, in the above embodiment, the control device 33 is equipped with a learning unit 77, and the neural network 90 is trained within the control device 33, but the configuration is not limited to this. For example, the learning unit 77 may be omitted, and the neural network 90 may be trained outside the control device 33, and the AI ​​model 101 (trained neural network 90) trained externally may be stored in the storage unit 57.

[0156] (g) In the above embodiment, the image frame W1 is square in shape satisfying m=n, but the image frame W1 may be rectangular in shape such that m≠n.

[0157] Furthermore, when obtaining training data and test image data, the image frame W1 may be omitted. In this case, a direction-common image may be used as the training data and test image data. [Explanation of Symbols]

[0158] 1...Printed circuit board, 3...Land, 5...Solder paste, 12a...Screen mask (mask), 12a1...Opening, 12b...Squeegee, 13...Solder inspection device, 32d...Camera (image data acquisition means), 75...Data processing unit (image data acquisition means, movement direction identification means), 78...Inspection unit (inspection image data acquisition means, reconstructed image data acquisition means, comparison means), 91...Encoder unit (encoding unit), 92...Decoder unit (decoding unit), 101...AI model (identification means), Hg1, Hg2, Hk1, Hk2, Hk3...Solder area images, G1, G2...Training data, K1, K2, K3...Inspection image data, Mg1, Mg2, Mk1, Mk2, Mk3...Direction common image, W1...Image frame.

Claims

1. A solder inspection apparatus for inspecting solder paste printed on a printed circuit board, comprising supplying solder paste to the upper surface of a mask having an opening, moving a predetermined squeegee in contact with the upper surface of the mask, and filling the opening with solder paste, Image data acquisition means capable of acquiring image data of a predetermined area to be inspected on the printed circuit board including printed solder paste, An identification means generated by training a neural network having an encoding unit that extracts features from input image data and a decoding unit that reconstructs image data from the features, using only image data related to good quality solder paste as training data, An inspection image data acquisition means that acquires inspection image data, including an image of the solder paste to be inspected, based on the image data acquired by the aforementioned image data acquisition means, A reconstructed image data acquisition means capable of inputting the aforementioned inspection image data into the identification means and acquiring the reconstructed image data as reconstructed image data, The system includes a comparison means capable of comparing the aforementioned inspection image data and the aforementioned reconstructed image data, Based on the comparison results obtained by the aforementioned comparison means, the system is configured to determine the quality of the solder paste. The aforementioned identification means is generated by training only with image data relating to solder paste where the direction of movement of the squeegee is common, Solder inspection apparatus characterized in that the reconstructed image data acquisition means is configured to input the inspection image data to the identification means such that the direction of movement of the squeegee in the learned learning data is the same as the direction of movement of the squeegee related to the solder paste to be inspected included in the inspection image data.

2. The aforementioned training data is obtained by setting a solder region image representing solder paste corresponding to one land, such that the direction of movement of the squeegee related to the solder paste is a predetermined direction, and then placing this direction common image in an image frame larger than the size of the direction common image. The solder inspection apparatus according to claim 1, characterized in that the inspection image data acquisition means acquires inspection image data obtained by setting the solder region image of the image data acquired by the image data acquisition means such that the direction of movement of the squeegee related to the solder paste in the solder region image is the same direction as the direction of movement of the squeegee in the learning data, and providing the inspection image data in an image frame of the same size as the image frame of the learning data.

3. The solder inspection apparatus according to claim 2, characterized in that the common orientation image in the learning data and the inspection image data is set so that the long side or short side extends along a predetermined direction while maintaining commonality with respect to the direction of movement of the squeegee.

4. The solder inspection apparatus according to claim 1, further comprising a movement direction determining means for determining the movement direction of the squeegee when printing the solder paste contained in the image data, based on the image data acquired by the image data acquisition means.

Images