Solder fillet inspecting device and solder fillet inspecting method

Abstract

The objective of the present invention is to reduce the effort and burden required to obtain an identification means serving as an AI model, and to enable the identification means to be commonly used even when the sizes of lands are different. Reconstructed image data are acquired by inputting inspection image data based on image data acquired by a camera 32d into AI models 101, 102 generated by training a specific neural network using only image data relating to non-defective solder fillets as training data, and the quality of a solder fillet is determined by comparing the inspection image data and the reconstructed image data. The training data are obtained by providing one solder fillet image in an image frame having a size larger than the size of the one solder fillet image. The inspection image data are obtained by providing one solder fillet image extracted from the image data acquired by the camera 32d in an image frame having the same size as the image frame of the training data.

Description

Solder fillet inspection device and solder fillet inspection method

[0001] The present invention relates to an inspection device and an inspection method for inspecting solder fillets used to solder electronic components.

[0002] Generally, in a board manufacturing line where electronic components are mounted on a printed circuit board, cream solder is first printed on the lands of the printed circuit board (solder printing process). Next, the electronic components are temporarily attached to the printed circuit board using the viscosity of the cream solder (mounting process). After that, the printed circuit board is introduced into a reflow furnace, where the cream solder is heated and melted to complete the soldering (reflow process). Such board manufacturing lines may be equipped with an inspection device that inspects the printed circuit board.

[0003] Recently, an inspection device using an AI model has been proposed for inspecting cream solder after a reflow process, i.e., the solder fillets used to solder electronic components (see, for example, Patent Document 1). This inspection device executes a predetermined inspection program to measure predetermined indices based on an inspection image and inspects the condition of the inspection target using the measurement values. The executed inspection program generates an AI model for each type (component type) of electronic component. Here, since the size of the lands on which electronic components are mounted often varies depending on the type of electronic component, this inspection program can be said to generate a different AI model for each land size.

[0004] Japanese Patent Application Laid-Open No. 2022-140951

[0005] However, since land sizes vary widely, generating and preparing different AI models for each land size requires complicated work, which may require a great deal of effort and time.

[0006] The present invention has been made in consideration of the above circumstances, and its purpose is to provide a solder fillet inspection device or the like that can reduce the effort and burden involved in obtaining an identification means as an AI model, and that allows the identification means to be used in common even when the land sizes are different.

[0007] The following describes each of the means suitable for achieving the above object, with specific effects of the corresponding means added as necessary.

[0008] Means 1. A solder fillet inspection device for inspecting solder fillets that solder electronic components on printed circuit boards, comprising: image data acquisition means capable of acquiring image data of a predetermined inspection area on the printed circuit board including the solder fillet; discrimination means generated by having a neural network having an encoding unit that extracts feature amounts from input image data and a decoding unit that reconstructs image data from the feature amounts learn only image data related to non-defective solder fillets as learning data; inspection image data acquisition means acquiring inspection image data including an image of the solder fillet to be inspected based on the image data acquired by the image data acquisition means; reconstructed image data acquisition means capable of inputting the inspection image data to the discrimination means and acquiring reconstructed image data as reconstructed image data; and comparison means capable of comparing the inspection image data and the reconstructed image data, and configured to be able to determine whether the solder fillet is good or bad based on the comparison result by the comparison means; the learning data comprises one solder fillet image showing a solder fillet corresponding to one land, arranged in an image frame larger than the size of the one solder fillet image, The solder fillet inspection device is characterized in that the inspection image data acquisition means acquires the inspection image data of the same size as the learning data, and the one solder fillet image extracted from the image data acquired by the image data acquisition means is placed in an image frame of the same size as the image frame of the learning data.

[0009] Note that one solder fillet image may be an image showing the entire connected component (lump portion) of the solder fillet, at least a part of which is located on one land (the image of means 2 described later), or an image showing only the part of the connected component of the solder fillet that is located on one land. Furthermore, one solder fillet image may include not only the connected component of the solder fillet, but also the lands and electrodes located around it.

[0010] In addition, one solder fillet image constituting the training data may be extracted from image data (actual image data) obtained by capturing an image of a printed circuit board on which a good solder fillet is provided (i.e., an image of an actual solder fillet), or may be a virtually generated image of a good solder fillet. Examples of the actual image data include image data accumulated in previous inspections and image data of good printed circuit boards visually selected by an operator after the reflow process.

[0011] Furthermore, the "neural network" includes, for example, a convolutional neural network having multiple convolutional layers. The "learning" includes, for example, deep learning. The "identification means (generative model)" includes, for example, an autoencoder and a convolutional autoencoder.

[0012] In addition, the "identification means" is generated by learning only image data relating to good solder fillets (learning data consisting of one solder fillet image relating to a good solder fillet arranged in an image frame). Therefore, the reconstructed image data generated when inspection image data relating to a defective solder fillet is input to the identification means will be approximately identical to the inspection image data in which the defective portion has been corrected (for example, the shape, area, etc. have been corrected). In other words, when a solder fillet has a defective portion, virtual image data relating to the solder fillet assuming that there is no defective portion is generated as the reconstructed image data relating to the solder fillet.

[0013] According to the above-mentioned means 1, the inspection image data is formed by placing one solder fillet image extracted from the image data acquired by the image data acquisition means in an image frame. Therefore, the size (width and height) of the inspection image data does not vary depending on the land size, but remains constant. This eliminates the need to prepare multiple different identification means for each land size, reducing the labor and time required to obtain the identification means. Furthermore, the identification means can be used in common even when the land sizes are different.

[0014] Furthermore, according to the above-mentioned means 1, the image frame of the learning data and the image frame of the test image data are the same size, and the sizes of the learning data and the test image data are the same. Therefore, when the test image data is input to the identification means, appropriate reconstructed image data corresponding to the test image data can be more reliably output, and ultimately the quality of the solder fillet can be more accurately determined. This makes it possible to more reliably obtain good inspection accuracy.

[0015] In addition, according to the above-described method 1, the inspection image data is compared with reconstructed image data obtained by inputting the inspection image data into the identification means and reconstructing the image data, and the quality of the solder fillet is determined based on the comparison results. Therefore, the two sets of compared image data relate to the same solder fillet. Therefore, unlike a method of determining quality by comparison with a separately prepared standard, it is not necessary to set relatively loose inspection conditions to prevent false detection, and stricter inspection conditions can be set. Furthermore, the imaging conditions of the printed circuit board to be inspected (e.g., the printed circuit board's placement position, placement angle, deflection, etc.) and the imaging conditions of the inspection device (e.g., lighting conditions, camera angle, etc.) can be matched between the two sets of compared image data. This combination allows for more accurate quality determination of the solder fillet.

[0016] Means 2. The solder fillet inspection device according to Means 1, further comprising a solder fillet image extraction means for extracting the one solder fillet image constituting the inspection image data from the image data acquired by the image data acquisition means, wherein the solder fillet image extraction means is capable of identifying an area occupied by a solder fillet in the image data acquired by the image data acquisition means, and extracting an image of a connected component in the identified area as the one solder fillet image constituting the inspection image data.

[0017] According to the above-mentioned means 2, the solder fillet image extraction means identifies the area occupied by the solder fillet in the image data acquired by the image data acquisition means, and extracts an image of the connected components in the identified area as one solder fillet image constituting the inspection image data. Therefore, one solder fillet image includes not only an image of the portion of the solder fillet located on the land, but also an image of the portion of the solder fillet that protrudes from the land. In other words, even if the solder fillet 5 protrudes from the land 3 as shown in Figure 27, one solder fillet image Ih will include this protruding portion as shown in Figure 28. This makes it possible to properly determine the quality of solder fillets that partially protrude from the land, further improving inspection accuracy.

[0018] Means 3. The solder fillet inspection device according to Means 1, further comprising second discrimination means generated by having a neural network having an encoding unit that extracts feature amounts from input image data and a decoding unit that reconstructs image data from the feature amounts learn only image data relating to solder fillets of non-defective products as second learning data, wherein the second learning data comprises the first solder fillet image arranged in a second image frame that is larger than the size of the first solder fillet image but smaller than the size of the image frame of the learning data, and when the size of the first solder fillet image extracted from the image data acquired by the image data acquisition means is smaller than the size of the second image frame, the inspection image data acquisition means acquires the inspection image data of the same size as the second learning data arranged in the second image frame, the reconstructed image data acquisition means inputs the inspection image data to the second discrimination means to acquire the reconstructed image data, and the comparison means is configured to compare the inspection image data and the reconstructed image data.

[0019] According to the above-mentioned means 3, when the size of one solder fillet image is relatively small, the inspection image data acquisition means acquires relatively small-sized inspection image data in which one solder fillet image is arranged in a relatively small-sized second image frame. The reconstructed image data acquisition means then inputs this relatively small inspection image data to the second identification means, which outputs the reconstructed image data. The comparison means compares the relatively small inspection image data and the reconstructed image data. Therefore, compared to when the image frame of the inspection image data is always a fixed size, the process for acquiring the reconstructed image data and the comparison process by the comparison means can be speeded up, thereby further improving the inspection speed.

[0020] Means 4. The solder fillet inspection device according to Means 1, wherein the learning data and the inspection image data are set so that the center or center of gravity of the one solder fillet image coincides with the center of the image frame, and the portion of the one solder fillet image on the electronic component side faces a predetermined direction.

[0021] The technical matters related to the above-mentioned means 4 may be applied to the above-mentioned means 3. That is, the second learning data may be set so that the center or center of gravity of one solder fillet image coincides with the center of the second image frame, and the part of the one solder fillet image on the electronic component side faces a predetermined direction. Of course, the same setting may be applied to the inspection image data related to the above-mentioned means 3.

[0022] According to the above-mentioned method 4, the orientation and position of the solder fillet in the training data and the inspection image data can be roughly aligned. Therefore, even if the training data used to generate the identification means is relatively small, the quality of the solder fillet can be determined with high accuracy. In other words, it is possible to obtain good inspection accuracy while more effectively reducing the labor and time required to obtain the identification means.

[0023] Means 5. The solder fillet inspection device according to Means 1, wherein the comparison means is configured to be able to compare the inspection image data and the reconstructed image data, with only the one solder fillet image in the inspection image data being used as a comparison object.

[0024] According to the above-mentioned means 5, the comparison means compares the inspection image data and the reconstructed image data, using only one solder fillet image in the inspection image data as the comparison target. In other words, when comparing the two image data, the comparison means does not compare any portion other than the one solder fillet image in the inspection image data. Therefore, compared to comparing the entirety of both image data, the processing load associated with comparing the two image data can be reduced, and inspection can be performed more quickly and efficiently. Furthermore, it is possible to more reliably prevent portions of the inspection image data other than the one solder fillet image, i.e., portions unrelated to the solder fillet, from affecting the pass / fail judgment, thereby further improving inspection accuracy.

[0025] Means 6. A solder fillet inspection method for inspecting solder fillets that solder electronic components on a printed circuit board, comprising: an image data acquisition step capable of acquiring image data of a predetermined inspection area on the printed circuit board including the solder fillet; an inspection image data acquisition step of acquiring inspection image data including an image of the solder fillet to be inspected based on the image data acquired in the image data acquisition step; a reconstructed image data acquisition step of inputting the inspection image data acquired in the inspection image data acquisition step into the identification means and acquiring reconstructed image data as reconstructed image data using a discrimination means generated by causing a neural network having an encoding unit that extracts feature amounts from input image data and a decoding unit that reconstructs image data from the feature amounts to learn only image data related to solder fillets of non-defective products; and a comparison step of comparing the inspection image data and the reconstructed image data, and judging the quality of the solder fillet based on the comparison result in the comparison step, wherein the learning data comprises one solder fillet image showing a solder fillet corresponding to one land, arranged in an image frame larger than the size of the one solder fillet image, A solder fillet inspection method characterized in that in the inspection image data acquisition process, the one solder fillet image extracted from the image data acquired by the image data acquisition process is placed in an image frame of the same size as the image frame of the learning data, and the inspection image data of the same size as the learning data is acquired.

[0026] According to the sixth aspect, the same effects as those of the first aspect can be achieved.

[0027] The technical matters relating to the above means may be combined as appropriate. For example, the technical matters relating to the above means 2 may be combined with the technical matters relating to the above means 3. Furthermore, for example, at least one of the technical matters relating to the above means 2 to 5 may be applied to the above means 6.

[0028] 1 is a partially enlarged plan view of a portion of a printed circuit board. FIG. 2 is a block diagram showing the configuration of a printed circuit board manufacturing line. FIG. 3 is a schematic configuration diagram showing a solder fillet inspection device. FIG. 4 is a block diagram showing the functional configuration of a solder fillet inspection device. FIG. 5 is a schematic diagram for explaining the structure of a neural network. FIG. 6 is a flowchart showing the flow of a learning process of a neural network. FIG. 7 is a flowchart showing the flow of an inspection process. FIG. 8 is a schematic diagram showing original learning image data. FIG. 9 is a schematic diagram showing an area occupied by a solder fillet in the original learning image data. FIG. 10 is a schematic diagram showing one solder fillet image extracted from the original learning image data. FIG. 11 is a schematic diagram showing a first image frame and first learning data Ga. FIG. 12 is a schematic diagram showing a first image frame and first learning data Gb. FIG. 13 is a schematic diagram showing a second image frame and second learning data Gc. FIG. 14 is a schematic diagram showing a second image frame and second learning data Gd. FIG. 15 is a schematic diagram showing an example of original inspection image data. FIG. 16 is a schematic diagram showing an area occupied by a solder fillet in the original inspection image data. FIG. 17 is a schematic diagram showing an example of one solder fillet image extracted from the original inspection image data. 1 is a schematic diagram showing an example of one solder fillet image extracted from the original inspection image data. FIG. 1 is a schematic diagram showing an example of first inspection image data Ka and a first image frame. FIG. 1 is a schematic diagram showing an example of first inspection image data Ka and a first image frame. FIG. 2 is a schematic diagram showing first inspection image data Kb and a first image frame. FIG. 3 is a schematic diagram showing an example of second inspection image data Kc and a second image frame. FIG. 4 is a schematic diagram showing an example of second inspection image data Kc and a second image frame. FIG. 5 is a schematic diagram showing second inspection image data Kd and a second image frame. FIG. 6 is a schematic diagram showing reconstructed image data output from a first AI model when first inspection image data Ka is input to the first AI model. FIG. 7 is a schematic diagram showing reconstructed image data output from a second AI model when second inspection image data Kc is input to the second AI model. FIG. 8 is a schematic diagram showing a solder fillet in a state where it protrudes from a land. FIG. 9 is a schematic diagram showing one solder fillet image relating to a solder fillet in a state where it protrudes from a land.

[0029] An embodiment of the present invention will now be described with reference to the accompanying drawings. First, the configuration of a printed circuit board will be described. Fig. 1 is an enlarged plan view of a portion of the printed circuit board.

[0030] 1, the printed circuit board 1 has a wiring pattern (not shown) made of copper foil and a plurality of lands 3 formed on the surface of a flat base substrate 2 made of glass epoxy resin or the like. A resist film 4 is coated on the surface of the base substrate 2 except for the lands 3.

[0031] Furthermore, electronic components 6 such as chips are mounted on the lands 3 via solder fillets 5. For convenience, in FIG. 1 and other figures, the solder fillets 5 are indicated by a dotted pattern. The solder fillets 5 are formed by heating cream solder, which is made by kneading solder particles with flux, in a reflow process described below. The solder fillets 5 are used to solder the electronic components 6 on the printed circuit board 1, and join the electrodes 6a to the lands 3. In this embodiment, the solder fillets 5 include relatively large solder fillets 5a and 5b provided on relatively large lands 3 and corresponding to the relatively large electronic components 6, and relatively small solder fillets 5c and 5d provided on relatively small lands 3 and corresponding to the relatively small electronic components 6.

[0032] Next, a manufacturing line (manufacturing process) for manufacturing the printed circuit board 1 will be described with reference to Fig. 2. Fig. 2 is a block diagram showing the configuration of a manufacturing line 10 for the printed circuit board 1. As shown in Fig. 2, the manufacturing line 10 is equipped with, in order from the upstream side (upper side in Fig. 2), a solder printer 12, a solder print state inspection device 13, a component mounter 14, a reflow device 15, and a solder fillet inspection device 16.

[0033] The solder printer 12 performs a solder printing process for printing cream solder on each land 3 of the printed circuit board 1. In the solder printing process, the cream solder is printed by, for example, screen printing. In screen printing, first, cream solder is supplied to the upper surface of a screen mask while the lower surface of the screen mask is in contact with the printed circuit board 1. The screen mask has a plurality of openings formed therein that correspond to each land 3 of the printed circuit board 1. Next, the openings are filled with cream solder by moving a predetermined squeegee while it is in contact with the upper surface of the screen mask. Thereafter, the printed circuit board 1 is separated from the lower surface of the screen mask, and the cream solder is printed on each land 3 of the printed circuit board 1.

[0034] The solder print state inspection device 13 inspects the shape of the cream solder printed on the land 3 and whether or not any foreign matter has adhered to the cream solder.

[0035] The component mounter 14 mounts the electronic component 6 on the lands 3 on which the cream solder has been printed. The component mounter 14 temporarily fixes the electrodes 6a of the electronic component 6 to the predetermined cream solder.

[0036] The reflow device 15 performs a reflow process in which the cream solder is heated and melted to solder the lands 3 to the electrodes 6a of the electronic component 6. The reflow process forms solder fillets 5 by heating the cream solder, and as a result, the lands 3 and the electronic component 6 (electrodes 6a) are fixed together.

[0037] The solder fillet inspection device 16 inspects whether the solder joint (soldering) has been properly performed in the reflow process by inspecting the shape, area, amount, etc. of the solder fillet 5. The solder fillet inspection device 16 will be described in detail later.

[0038] In addition, although not shown, the production line 10 is equipped with conveyors or the like for transporting the printed circuit board 1 between the above-mentioned devices, such as between the solder printer 12 and the solder print state inspection device 13. Furthermore, branching devices are provided between the solder print state inspection device 13 and the component mounter 14 and downstream of the solder fillet inspection device 16. The printed circuit boards 1 that have been determined to be non-defective by the solder print state inspection device 13 or the solder fillet inspection device 16 are guided directly downstream, while the printed circuit boards 1 that have been determined to be defective are discharged by the branching devices to a defective product storage area.

[0039] Next, the configuration of the solder fillet inspection device 16 will be described in detail with reference to Figures 3 and 4. Figure 3 is a schematic diagram showing the configuration of the solder fillet inspection device 16. Figure 4 is a block diagram showing the functional configuration of the solder fillet inspection device 16.

[0040] The solder fillet inspection device 16 includes a transport mechanism 31 for transporting and positioning the printed circuit board 1, an inspection unit 32 for obtaining image data of the printed circuit board 1, and a control device 33 (see Figure 4) for controlling the drive of the transport mechanism 31 and the inspection unit 32, as well as performing various controls, image processing, and arithmetic processing in the solder fillet inspection device 16.

[0041] The transport mechanism 31 includes a pair of transport rails 31a arranged along the direction in which the printed circuit board 1 is carried in and out, and an endless conveyor belt 31b rotatably disposed on each of the transport rails 31a. Although not shown, the transport mechanism 31 also includes a driving means such as a motor for driving the conveyor belt 31b, and a chucking mechanism for positioning the printed circuit board 1 at a predetermined position. The transport mechanism 31 is driven and controlled by a control device 33 (a transport mechanism control unit 79 described later).

[0042] With the above configuration, the printed circuit board 1 is carried into the solder fillet inspection device 16. Both side edges in the width direction perpendicular to the carrying-in / out direction are inserted into the conveyor rails 31a, and the printed circuit board 1 is placed on the conveyor belt 31b. The conveyor belt 31b then starts operating, transporting the printed circuit board 1 to a predetermined inspection position. When the printed circuit board 1 reaches the inspection position, the conveyor belt 31b stops and the chucking mechanism is activated. The operation of this chucking mechanism pushes up the conveyor belt 31b, and both side edges of the printed circuit board 1 are clamped between the conveyor belt 31b and the upper edge of the conveyor rail 31a. This positions and fixes the printed circuit board 1 at the inspection position. When the inspection is completed, the chucking mechanism releases the fixation, and the conveyor belt 31b starts operating. The printed circuit board 1 is then carried out of the solder fillet inspection device 16. Of course, the configuration of the transport mechanism 31 is not limited to the above, and other configurations may be employed.

[0043] The inspection unit 32 is disposed above the transport rail 31a (the transport path for the printed circuit board 1). The inspection unit 32 includes a first lighting device 32a, a second lighting device 32b, a third lighting device 32c, and a camera 32d. In this embodiment, the camera 32d constitutes the "image data acquisition means."

[0044] The inspection unit 32 also includes an X-axis movement mechanism 32e (see FIG. 4) that allows movement in the X-axis direction (left-right direction in FIG. 3), and a Y-axis movement mechanism 32f (see FIG. 4) that allows movement in the Y-axis direction (front-back direction in FIG. 3). These movement mechanisms 32e and 32f are driven and controlled by the control device 33 (a movement mechanism control section 76, which will be described later).

[0045] When performing three-dimensional measurement of the printed circuit board 1, the first illumination device 32a and the second illumination device 32b each irradiate a predetermined inspection area on the printed circuit board 1 from diagonally above with predetermined light for three-dimensional measurement (pattern light having a striped light intensity distribution).

[0046] Specifically, the first lighting device 32a includes a first light source 32a1 that emits a predetermined light, and a first liquid crystal shutter 32a2 that forms a first grating that converts the light from the first light source 32a1 into a first pattern light having a striped light intensity distribution, and is driven and controlled by the control device 33 (the lighting control unit 72 described later).

[0047] The second lighting device 32b includes a second light source 32b1 that emits a predetermined light, and a second liquid crystal shutter 32b2 that forms a second grating that converts the light from the second light source 32b1 into a second pattern light having a striped light intensity distribution, and is driven and controlled by a control device 33 (an illumination control unit 72 described later).

[0048] With the above configuration, the light emitted from each light source 32a1, 32b1 is guided to a condenser lens (not shown), where it is converted into parallel light, and then guided to a projection lens (not shown) via liquid crystal shutters 32a2, 32b2, and projected as patterned light onto printed circuit board 1. In this embodiment, switching control of liquid crystal shutters 32a2, 32b2 is performed so that the phase of each patterned light is shifted by a quarter pitch.

[0049] Furthermore, by using the liquid crystal shutters 32a2 and 32b2 as the grating, it is possible to irradiate pattern light that is close to an ideal sine wave. This improves the measurement resolution of three-dimensional measurement. In addition, the phase shift of the pattern light can be electrically controlled, which allows for a more compact device.

[0050] When performing two-dimensional measurement of the printed circuit board 1, the third illumination device 32c irradiates a predetermined inspection area on the printed circuit board 1 with predetermined light for two-dimensional measurement (e.g., uniform light). The third illumination device 32c is equipped with a ring light capable of irradiating blue light, a ring light capable of irradiating green light, and a ring light capable of irradiating red light. Note that the third illumination device 32c has a configuration similar to that of known technology, and therefore detailed description thereof will be omitted.

[0051] The camera 32d captures an image of a predetermined inspection area of ​​the printed circuit board 1 from directly above. The camera 32d has an imaging element such as a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor, and an optical system (lens unit, diaphragm, etc.) that forms an image of the printed circuit board 1 on the imaging element, and is arranged so that its optical axis is aligned in the vertical direction (Z-axis direction). Of course, the imaging element is not limited to these, and other imaging elements may be used.

[0052] The camera 32d is driven and controlled by the control device 33 (camera control unit 73, described later). More specifically, the control device 33 executes an image capturing process by the camera 32d in synchronization with the irradiation process by each of the lighting devices 32a, 32b, and 32c. As a result, light irradiated from any of the lighting devices 32a, 32b, and 32c and reflected by the printed circuit board 1 is captured by the camera 32d. As a result, image data of the inspection area of ​​the printed circuit board 1, including the solder fillet 5, is acquired. Note that the "inspection area" of the printed circuit board 1 is one area out of multiple areas preset on the printed circuit board 1, with the size of the imaging field of view (imaging range) of the camera 32d being one unit.

[0053] Furthermore, the camera 32d in this embodiment is configured as a color camera, which makes it possible to simultaneously capture images of the light of each color that is simultaneously irradiated from the ring lights of each color of the third illumination device 32c and reflected by the printed circuit board 1.

[0054] The image data captured and generated by the camera 32d is converted into a digital signal inside the camera 32d and then transferred in the form of a digital signal to the control device 33 (an image acquisition unit 74 described below). The control device 33 then stores the transferred image data and performs various image processing, arithmetic processing, and the like based on the image data.

[0055] The control device 33 is composed of a computer including a CPU (Central Processing Unit) that executes predetermined arithmetic processing, a ROM (Read Only Memory) that stores various programs and fixed value data, a RAM (Random Access Memory) that temporarily stores various data when executing various arithmetic processing, and peripheral circuits for these.

[0056] The control device 33 functions as various functional units such as a main control unit 71, a lighting control unit 72, a camera control unit 73, an image acquisition unit 74, a data processing unit 75, a movement mechanism control unit 76, a learning unit 77, an inspection unit 78, and a transport mechanism control unit 79, as the CPU operates in accordance with various programs.

[0057] However, the various functional units are realized by the cooperation of various hardware such as the CPU, ROM, RAM, etc., and there is no need to clearly distinguish between functions realized by hardware and functions realized by software, and some or all of these functions may be realized by hardware circuits such as ICs.

[0058] Furthermore, the control device 33 is provided with an input unit 55 consisting of a keyboard, mouse, touch panel, etc., a display unit 56 with a display screen consisting of a liquid crystal display, etc., a memory unit 57 capable of storing various data, programs, calculation results, test results, etc., and a communication unit 58 capable of sending and receiving various data to and from the outside.

[0059] Here, the above-mentioned various functional units that constitute the control device 33 will be described in detail.

[0060] The main control unit 71 is a functional unit that controls the entire solder fillet inspection device 16 and is configured to be able to send and receive various signals to and from other functional units such as the illumination control unit 72 and camera control unit 73 .

[0061] The illumination control unit 72 is a functional unit that controls the driving of the illumination devices 32 a , 32 b , and 32 c , and performs switching control of the illumination light based on a command signal from the main control unit 71 .

[0062] The camera control unit 73 is a functional unit that controls the driving of the camera 32 d, and controls the timing of image capture based on a command signal from the main control unit 71 .

[0063] The image acquisition unit 74 is a functional unit for capturing image data captured and acquired by the camera 32d.

[0064] The data processing unit 75 is a functional unit that performs predetermined image processing on the image data captured by the image acquisition unit 74, and performs two-dimensional measurement processing, three-dimensional measurement processing, and the like using the image data.

[0065] The movement mechanism control unit 76 is a functional unit that drives and controls the X-axis movement mechanism 32 e and the Y-axis movement mechanism 32 f, and controls the position of the inspection unit 32 based on a command signal from the main control unit 71. By controlling and driving the X-axis movement mechanism 32 e and the Y-axis movement mechanism 32 f, the movement mechanism control unit 76 can move the inspection unit 32 to a position above any inspection area of ​​the printed circuit board 1 that is positioned and fixed at the inspection position. Then, the inspection unit 32 is moved sequentially to a plurality of inspection areas set on the printed circuit board 1, and inspections of the inspection areas are performed, thereby inspecting the entire printed circuit board 1.

[0066] The learning unit 77 is a functional unit that uses learning data to learn a deep neural network 90 (hereinafter simply referred to as the "neural network 90"; see Figure 5) and constructs a first AI (Artificial Intelligence) model 101 as an "identification means" and a second AI model 102 as a "second identification means."

[0067] As will be described later, each AI model 101, 102 in this embodiment is a generative model constructed by deep learning a neural network 90 using only image data relating to good solder fillets 5 as training data, and has the structure of a so-called autoencoder.

[0068] The structure of the neural network 90 will now be described with reference to Fig. 5. Fig. 5 is a schematic diagram conceptually illustrating the structure of the neural network 90. ​​As shown in Fig. 5, the neural network 90 has a convolutional auto-encoder (CAE) structure that includes an encoder unit 91 serving as an "encoding unit" that extracts a feature (latent variable) TA from input image data GA, and a decoder unit 92 serving as a "decoding unit" that reconstructs image data GB from the feature TA.

[0069] The structure of a convolutional autoencoder is well known, and therefore a detailed description thereof will be omitted. However, the encoder unit 91 has a plurality of convolution layers 93, and in each convolution layer 93, a convolution operation is performed on input data using a plurality of filters (kernels) 94, and the result is output as input data for the next layer. Similarly, the decoder unit 92 has a plurality of deconvolution layers 95, and in each deconvolution layer 95, a deconvolution operation is performed on input data using a plurality of filters (kernels) 96, and the result is output as input data for the next layer. Then, in the learning process described below, the weights (parameters) of each filter 94, 96 are updated.

[0070] The inspection unit 78 is a functional unit that inspects the solder fillet 5. In this embodiment, the inspection unit 78 inspects whether the solder fillet 5 is properly formed in terms of shape, area, quantity, and the like.

[0071] The transport mechanism control unit 79 is a functional unit that controls the drive of the transport mechanism 31 , and controls the position of the printed circuit board 1 based on a command signal from the main control unit 71 .

[0072] The memory unit 57 is composed of a hard disk drive (HDD) or a solid state drive (SSD), and has a predetermined memory area for storing, for example, each AI model 101, 102 (neural network 90 and learning information acquired by its learning).

[0073] The communication unit 58 includes a wireless communication interface conforming to communication standards such as a wired LAN (Local Area Network) or a wireless LAN, and is configured to be able to transmit and receive various data to and from the outside. For example, the results of the inspection performed by the inspection unit 78 are output to the outside via the communication unit 58, and the results of the inspection performed by the solder print state inspection device 13 are input via the communication unit 58.

[0074] Next, the learning process of the neural network 90 performed by the solder fillet inspection device 16 will be described with reference to the flowchart of FIG.

[0075] When the learning process is started based on the execution of a predetermined learning program, the main control unit 71 first performs pre-processing for learning the neural network 90 in step S101.

[0076] In this pre-processing, first, inspection information of a large number of printed circuit boards 1 stored in advance in the solder fillet inspection device 16 is acquired. Next, based on the inspection information, original learning image data Ig, which is image data relating to solder fillets 5 of non-defective products that have passed the post-reflow inspection, is acquired from the storage unit 57 (see, for example, FIG. 8 ).

[0077] This training original image data Ig relates to the printed circuit board 1 after the reflow process, and is used to obtain training data Ga, Gb, Gc, and Gd (hereinafter, sometimes abbreviated as "training data Ga-Gd") to be described later, which are used for training the neural network 90. ​​The training original image data Ig also includes three-dimensional data, which is image data obtained by imaging the printed circuit board 1 with the camera 32d while irradiating it with patterned light from the first illumination device 32a or the second illumination device 32b, and two-dimensional data, which is image data obtained by imaging the printed circuit board 1 with the camera 32d while irradiating it with uniform light from the third illumination device 32c.

[0078] The learning original image data Ig may be image data obtained by the camera 32d without any special processing (e.g., monochromatic luminance image data or RGB luminance image data), or image data obtained by performing predetermined processing on the image data obtained by the camera 32d (e.g., HLS image data obtained by converting RGB image data, or height image data obtained by converting image data). Furthermore, the learning original image data Ig may be image data obtained when using a so-called color highlight method (a method in which red, green, and blue light are irradiated onto the surface of the solder fillet 5 at different incident angles, and the reflected light of each color is photographed by the camera 32d to detect the three-dimensional shape of the solder fillet 5 as two-dimensional hue information).

[0079] Next, first training data Ga, Gb and second training data Gc, Gd are created from the acquired original training image data Ig (see FIGS. 11 to 14 for the training data Ga to Gd). In this embodiment, the first training data Ga, Gb correspond to "training data." The first training data Ga, Gb are used to generate the first AI model 101, and the second training data Gc, Gd are used to generate the second AI model 102.

[0080] To obtain each of the learning data Ga to Gd, first, the area occupied by the solder fillet 5 in the acquired learning original image data Ig is identified (see, for example, FIG. 9 ). If the learning original image data Ig is two-dimensional data, the area occupied by the solder fillet 5 is identified using, for example, brightness, hue, saturation, etc. If the learning original image data Ig is three-dimensional data, the area occupied by the solder fillet 5 is identified using, for example, height information, etc.

[0081] Next, an image of a connected component (a lump portion) in the area occupied by the identified solder fillet 5 is extracted as one solder fillet image Ih (see, for example, FIG. 10 ). One solder fillet image Ih corresponds to one land 3, and in this embodiment, a connected component (a lump portion) located on one land 3 in the design within the area occupied by the solder fillet 5 is extracted as one solder fillet image Ih. Note that one connected component (a lump portion) may simply be extracted as one solder fillet image Ih without considering the position of the land 3.

[0082] Next, an image frame into which the extracted solder fillet image Ih is to be pasted is selected from the first image frame W1 and the second image frame W2 (see FIGS. 11 to 14 for the image frames W1 and W2).

[0083] In this embodiment, the first image frame W1 is a rectangular image with a height (width in the vertical direction of the paper in Figure 11, etc.) of n (pixels) and a width (width in the horizontal direction of the paper in Figure 11, etc.) of n (pixels), and its size (width and height) is set based on design data, etc. to be larger than the size of the extracted solder fillet image Ih.

[0084] The second image frame W2 is a rectangular image with a height of m (pixels) and a width of m (pixels), and its size (width and height) is set to be smaller than the size of the first image frame W1.

[0085] Furthermore, all pixels in each image frame W1, W2 (when nothing is pasted) have the same value. For example, the brightness and height information of each pixel that makes up each image frame W1, W2 is set to "0." It is preferable that this same value be relatively different from the value of the constituent pixels of the solder fillet image Ih of 1. Therefore, it is preferable to use a value other than a normally used value (for example, a negative number) as this same value.

[0086] The selection of the image frames W1 and W2 is based on the size and shape of the first solder fillet image Ih. If the size (width and height) of the first solder fillet image Ih is larger than the size of the second image frame W2, the first image frame W1 is selected. On the other hand, if the size of the first solder fillet image Ih is smaller than the size of the second image frame W2, the second image frame W2 is selected.

[0087] Next, one solder fillet image Ih is pasted onto the selected image frames W1 and W2, thereby obtaining each of the learning data Ga to Gd in which one solder fillet image Ih is provided in the image frames W1 and W2.

[0088] In this embodiment, first learning data Ga (see, for example, Figure 11) is obtained by pasting one solder fillet image Ih corresponding to solder fillet 5a into the first image frame W1, and first learning data Gb (see, for example, Figure 12) is obtained by pasting one solder fillet image Ih corresponding to solder fillet 5b into the first image frame W1.

[0089] Furthermore, by pasting one solder fillet image Ih corresponding to solder fillet 5c into the second image frame W2, second learning data Gc (see, for example, Figure 13) is obtained, and by pasting one solder fillet image Ih corresponding to solder fillet 5d into the second image frame W2, second learning data Gd (see, for example, Figure 14) is obtained.

[0090] By adjusting the pasting position and rotating the first solder fillet image Ih, each of the learning data Ga to Gd is set so that the center or center of gravity of the first solder fillet image Ih coincides with the center of the image frames W1 and W2, and the portion of the first solder fillet image Ih on the side of the electronic component 6 faces a predetermined direction. The "portion of the first solder fillet image Ih on the side of the electronic component 6" refers to the portion of the first solder fillet image Ih adjacent to the electronic component 6, which is linear in this embodiment. The first solder fillet image Ih is then rotated so that this linear portion faces a predetermined direction (for example, to the right).

[0091] Then, by repeating the above process of extracting one solder fillet image Ih and pasting one solder fillet image Ih into the selected image frames W1 and W2, each of the training data Ga to Gd is obtained from one training original image data Ig. Furthermore, by using multiple training original image data Ig, the required number of first training data Ga, Gb and second training data Gc, Gd are finally obtained. In this embodiment, each of the training data Ga to Gd includes data obtained based on two-dimensional data and data obtained based on three-dimensional data.

[0092] In step S101, the number of pieces of learning data Ga to Gd required for learning is acquired. In the following step S102, the learning unit 77 prepares an unlearned neural network 90 based on a command from the main control unit 71. For example, the learning unit 77 reads out the neural network 90 stored in advance in the storage unit 57 or the like. Alternatively, the neural network 90 is constructed based on network configuration information (e.g., the number of layers of the neural network, the number of nodes in each layer, etc.) stored in the storage unit 57 or the like.

[0093] In this embodiment, two neural networks 90 are constructed: one for performing learning using the first learning data Ga and Gb, and one for performing learning using the second learning data Gc and Gd. Furthermore, two neural networks 90 are constructed: one for performing learning using the learning data Ga to Gd acquired based on two-dimensional data, and one for performing learning using the learning data Ga to Gd acquired based on three-dimensional data. Therefore, in this embodiment, a total of four neural networks 90 are constructed.

[0094] In step S103, reconstructed image data is acquired. That is, based on a command from the main control unit 71, the learning unit 77 provides the training data Ga-Gd acquired in step S102 as input data to the input layer of the neural network 90, thereby acquiring reconstructed image data output from the output layer of the neural network 90. ​​More specifically, the learning unit 77 provides the training data Ga-Gd acquired in step S102 that corresponds to the neural network 90 as input data to the input layer of the neural network 90, thereby acquiring reconstructed image data output from the output layer of the neural network 90. ​​For example, the learning unit 77 provides the first training data Ga and Gb obtained from two-dimensional data as input data to the input layer of the neural network 90 that performs training using the first training data Ga and Gb, and acquires the reconstructed image data output from the neural network 90. ​​That is, the learning unit 77 inputs appropriate training data Ga-Gd to each of the four types of neural networks 90, and acquires the output reconstructed image data.

[0095] In the following step S104, the learning unit 77 compares the input learning data Ga to Gd with the reconstructed image data output by the neural network 90, and determines whether the error is sufficiently small (whether it is below a predetermined threshold).

[0096] If the error is sufficiently small, in step S106, the learning unit 77 determines whether the learning termination condition is met. For example, if a positive determination is made in step S104 a predetermined number of times in succession without going through the processing of step S105 described below, or if learning using all of the prepared learning data Ga to Gd is repeated a predetermined number of times, it is determined that the termination condition is met. If the termination condition is met, the neural network 90 and its learning information (such as updated parameters described below) are stored in the storage unit 57 as AI models 101 and 102, and the learning process is terminated.

[0097] In this embodiment, the first AI model 101 ultimately stores an AI model obtained by learning the first learning data Ga, Gb obtained from two-dimensional data, and an AI model obtained by learning the first learning data Ga, Gb obtained from three-dimensional data.

[0098] In addition, as the second AI model 102, an AI model obtained by learning the second learning data Gc, Gd obtained from two-dimensional data and an AI model obtained by learning the second learning data Gc, Gd obtained from three-dimensional data are stored.

[0099] On the other hand, if the termination condition is not met in step S106, the process returns to step S102, and the neural network 90 is trained again.

[0100] If the error is not sufficiently small in step S104, the network is updated (the neural network 90 is trained) in step S105, and then the process returns to step S103 to repeat the above series of processes.

[0101] Specifically, in the network update process of step S105, a known learning algorithm such as backpropagation is used to update the weights (parameters) of the filters 94, 96 in the neural network 90 to more appropriate ones so that a loss function representing the difference between the training data Ga to Gd and the reconstructed image data is minimized. Note that, for example, BCE (Binary Cross-entropy) can be used as the loss function.

[0102] By repeating the processes of steps S103 to S105 many times, the neural network 90 minimizes the error between the learning data Ga to Gd and the reconstructed image data, and outputs more accurate reconstructed image data.

[0103] When image data relating to a good solder fillet 5 is input, each of the finally obtained AI models 101, 102 generates reconstructed image data that approximately matches the image data. Furthermore, when image data relating to a defective solder fillet 5 in terms of shape, area, or quantity is input, each of the AI ​​models 101, 102 generates reconstructed image data that approximately matches the image data obtained by correcting the shape, area, and quantity of the solder fillet 5. In other words, when the solder fillet 5 is defective, virtual image data relating to the solder fillet 5, assuming that there is no defective portion, is generated as the reconstructed image data relating to the solder fillet 5.

[0104] Next, the inspection process performed by the solder fillet inspection device 16 will be described with reference to the flowchart of Fig. 7. However, the inspection process shown in Fig. 7 is a process that is executed for each area to be inspected on the printed circuit board 1.

[0105] When the printed circuit board 1 is carried into the solder fillet inspection device 16 and positioned at a predetermined inspection position, the inspection process starts based on the execution of a predetermined inspection program.

[0106] When the inspection process starts, an image data acquisition process is first performed in step S301. In the image data acquisition process, original inspection image data Ik (see, for example, FIG. 15 ) related to the printed circuit board 1 to be inspected is acquired. The original inspection image data Ik is image data for obtaining inspection image data Ka, Kb, Kc, and Kd (hereinafter, sometimes abbreviated as "inspection image data Ka-Kd"), which will be described later. Note that in this embodiment, an example of the printed circuit board 1 to be inspected is one in which the solder fillet 5 protrudes from the land 3 or the solder fillet 5 has an abnormality in shape or area.

[0107] The original inspection image data Ik includes three-dimensional data, which is image data obtained by imaging the printed circuit board 1 with the camera 32d while irradiating it with patterned light from the first illumination device 32a or the second illumination device 32b, and two-dimensional data, which is image data obtained by imaging the printed circuit board 1 with the camera 32d while irradiating it with uniform light from the third illumination device 32c. In the image data acquisition process, a three-dimensional data acquisition process and a two-dimensional data acquisition process are performed.

[0108] First, the three-dimensional data acquisition process will be described. In this process, the phase of the first pattern light irradiated from the first illumination device 32a is changed and four imaging processes are performed under the first pattern light with different phases. Then, the phase of the second pattern light irradiated from the second illumination device 32b is changed and four imaging processes are performed under the second pattern light with different phases, thereby acquiring a total of eight sets of three-dimensional data. This will be described in detail below.

[0109] As described above, when the printed circuit board 1 brought into the solder fillet inspection device 16 is positioned and fixed at a predetermined inspection position, based on instructions from the main control unit 71, the movement mechanism control unit 76 first drives and controls the X-axis movement mechanism 32e and the Y-axis movement mechanism 32f to move the inspection unit 32, and aligns the imaging field of view (imaging range) of the camera 32d with the predetermined inspection area of ​​the printed circuit board 1.

[0110] Additionally, the illumination control unit 72 controls the switching of the liquid crystal shutters 32a2, 32b2 of the two illumination devices 32a, 32b, and sets the positions of the first grating and the second grating formed on the two liquid crystal shutters 32a2, 32b2 to predetermined reference positions.

[0111] After the first and second gratings are switched and set, the illumination control unit 72 causes the first light source 32a1 of the first illumination device 32a to emit light and irradiate the first pattern light, and the camera control unit 73 drives and controls the camera 32d to perform a first imaging process under the first pattern light. The image data generated by the imaging process is continuously imported into the image acquisition unit 74 (and so on). This allows three-dimensional data of the inspection area including the multiple lands 3 (solder fillets 5) to be acquired.

[0112] Thereafter, upon completion of the first imaging process under the first pattern light, the illumination control unit 72 turns off the first light source 32a1 of the first illumination device 32a and executes a switching process for the first liquid crystal shutter 32a2. Specifically, the illumination control unit 72 switches the position of the first grating formed on the first liquid crystal shutter 32a2 from the reference position to a second position where the phase of the first pattern light is shifted by a quarter pitch (90°).

[0113] When the first grating setting is complete, the illumination control unit 72 causes the light source 32a1 of the first illumination device 32a to emit light and irradiate the first pattern light, and the camera control unit 73 drives and controls the camera 32d to perform a second imaging process under the first pattern light. Thereafter, by repeating the same process, four sets of three-dimensional data are acquired under the first pattern light, each with a phase difference of 90°.

[0114] Next, the lighting control unit 72 causes the second light source 32b1 of the second lighting device 32b to emit light and irradiate the second pattern light, and the camera control unit 73 drives and controls the camera 32d to perform the first imaging process under the second pattern light.

[0115] Thereafter, at the same time as the completion of the first imaging process under the second pattern light, the illumination control unit 72 turns off the second light source 32b1 of the second illumination device 32b and executes a switching process for the second liquid crystal shutter 32b2. Specifically, the position of the second grating formed on the second liquid crystal shutter 32b2 is switched from the reference position to a second position where the phase of the second pattern light is shifted by a quarter pitch (90°).

[0116] When the second grating setting is complete, the illumination control unit 72 causes the light source 32b1 of the second illumination device 32b to emit light and irradiate the second pattern light, and the camera control unit 73 drives and controls the camera 32d to perform a second imaging process under the second pattern light. Thereafter, by repeating the same process, four sets of three-dimensional data are acquired under the second pattern light, each with a phase difference of 90°.

[0117] Next, the two-dimensional data acquisition process will be described. In this process, based on a command from the main control unit 71, the illumination control unit 72 causes the third illumination device 32c to emit light, irradiating a predetermined inspection area with uniform light, while the camera control unit 73 drives and controls the camera 32d to perform an image capture process under the uniform light. As a result, the predetermined inspection area on the printed circuit board 1 is imaged, and two-dimensional data related to the inspection area is acquired.

[0118] The acquired original inspection image data Ik (three-dimensional data and two-dimensional data) is stored in the storage unit 57.

[0119] The original inspection image data Ik may be image data obtained by the camera 32d without any special processing (for example, monochrome luminance image data or RGB luminance image data), or image data obtained by performing predetermined processing on the image data obtained by the camera 32d (for example, HLS image data obtained by converting RGB image data, height image data obtained by converting image data, etc.). Furthermore, the original inspection image data Ik may be image data obtained when using a so-called color highlight method.

[0120] Next, in step S302, a test image data acquisition step is executed, in which first test image data Ka, Kb and second test image data Kc, Kd (described later) are acquired based on the original test image data Ik obtained in the image data acquisition step.

[0121] To obtain these inspection image data Ka to Kd, first, the area occupied by the solder fillet 5 in the acquired original inspection image data Ik is identified (see, for example, FIG. 16). The area occupied by the solder fillet 5 can be identified using information such as brightness, hue, saturation, and height.

[0122] Next, a single solder fillet image Ih (see, for example, FIGS. 17 and 18 ) is extracted from the region occupied by the identified solder fillet 5. The single solder fillet image Ih corresponds to a single land 3. In this embodiment, the entire image of the connected components (lump portions) in the identified region that at least partially overlap with the single land 3 in the design data or manufacturing data is extracted as the single solder fillet image Ih. Therefore, if a portion of a connected component extends beyond the land 3 in the design data or manufacturing data, the single solder fillet image Ih is composed of the entire connected component, including the portion that extends beyond the land 3 (see, for example, FIG. 18 ). Note that a single connected component may be simply extracted as the single solder fillet image Ih without using design data, etc. Alternatively, the single solder fillet image Ih may include not only the connected component (lump portion) of the solder fillet 5 but also the land 3, electrode 6a, and other components located around it. In this embodiment, the inspection unit 78 that extracts one solder fillet image Ih from the original inspection image data Ik constitutes the "solder fillet image extraction means."

[0123] Then, based on the size and shape of the extracted solder fillet image Ih, an appropriate image frame is selected from the first image frame W1 and the second image frame W2, and inspection image data Ka to Kd are obtained by placing the solder fillet image Ih in the selected image frames W1 and W2 (see Figures 19 to 24 for inspection image data Ka to Kd).

[0124] That is, if the size of one extracted solder fillet image Ih is larger than the size of the second image frame W2, the first inspection image data Ka, Kb (see, for example, Figures 19 to 21) are obtained by pasting the one solder fillet image Ih into the first image frame W1. Therefore, the size (width and height) of the first inspection image data Ka, Kb is the same as the size of the first learning data Ga, Gb. In Figure 20, for reference, a solder fillet 5 with a normal shape and area is virtually shown by a two-dot chain line.

[0125] Furthermore, if the size of the extracted solder fillet image Ih is smaller than the size of the second image frame W2, the second inspection image data Kc, Kd (see, for example, FIGS. 22 to 24) are obtained by pasting the extracted solder fillet image Ih into the second image frame W2. Therefore, the size of the second inspection image data Kc, Kd is the same as the size of the second learning data Gc, Gd.

[0126] When pasting one solder fillet image Ih into the image frames W1 and W2, adjustment of the pasting position, image rotation, etc. are performed. As a result, in each of the inspection image data Ka to Kd, similar to each of the learning data Ga to Gd, the center or center of gravity of one solder fillet image Ih coincides with the center of the image frames W1 and W2, and the portion of the one solder fillet image Ih on the electronic component 6 side faces a predetermined direction.

[0127] Then, by repeating the above process of extracting one solder fillet image Ih and pasting one solder fillet image Ih into the selected image frames W1 and W2, each of the inspection image data Ka-Kd is acquired from one of the original inspection image data Ik. In this embodiment, each of the inspection image data Ka-Kd includes data acquired based on two-dimensional data and data acquired based on three-dimensional data. In this embodiment, the inspection unit 78 that acquires the inspection image data Ka-Kd constitutes the "inspection image data acquisition means."

[0128] In the following step S303, a reconstructed image data acquisition step is executed. Specifically, based on a command from the main control unit 71, the inspection unit 78 inputs the inspection image data Ka-Kd acquired in step S302 to the input layer of the AI ​​models 101 and 102 corresponding to the type of the inspection image data Ka-Kd. Therefore, the first inspection image data Ka and Kb are input to the first AI model 101, and the second inspection image data Kc and Kd are input to the second AI model 102. Furthermore, the inspection image data Ka-Kd acquired based on two-dimensional data are input to the AI ​​models 101 and 102 corresponding to the two-dimensional data, and the inspection image data Ka-Kd acquired based on three-dimensional data are input to the AI ​​models 101 and 102 corresponding to the three-dimensional data. Then, the image data reconstructed by the AI ​​models 101 and 102 and output from the output layer is acquired as reconstructed image data. The acquired reconstructed image data is stored in association with the test image data Ka to Kd from which the reconstructed image data was derived.

[0129] Here, when each AI model 101, 102 receives inspection image data Ka, Kc (see Figures 20 and 22) relating to a solder fillet 5 with an inappropriate shape, etc., it outputs image data S relating to a good solder fillet 5 whose shape, etc. has been corrected, as reconstructed image data S, based on the learning described above (see, for example, Figures 25 and 26).

[0130] On the other hand, when inspection image data Ka to Kd relating to a non-defective solder fillet 5 are input, each of the AI ​​models 101 and 102 outputs image data relating to the non-defective solder fillet 5 that is substantially identical to the inspection image data Ka to Kd as reconstructed image data S. The size (width and height) of the reconstructed image data S is the same as the size of the inspection image data Ka to Kd from which it is derived. In this embodiment, the inspection unit 78 that acquires the reconstructed image data S constitutes the "reconstructed image data acquisition means."

[0131] In step S304, a quality determination process is performed based on the acquired reconstructed image data S. In the quality determination process, based on instructions from the main control unit 71, the inspection unit 78 compares the entire test image data Ka-Kd acquired in step S302 with the entire reconstructed image data S acquired in step S303 using the test image data Ka-Kd, and calculates the difference between the two sets of image data Ka-Kd and S. For example, dots (pixels) at the same coordinates in the two sets of image data Ka-Kd and S are compared, and the area (number of dots) of a cluster of dots whose brightness difference is equal to or greater than a predetermined value is calculated. In this embodiment, the inspection unit 78, which compares the test image data Kd-Kd and the reconstructed image data S, constitutes a "comparison means." Furthermore, the process of comparing the test image data Ka-Kd and the reconstructed image data S corresponds to a "comparison process."

[0132] Next, the inspection unit 78 determines whether the calculated difference is greater than a predetermined threshold value. If the calculated difference is greater than the predetermined threshold value, the inspection unit 78 determines the product as a "good product," whereas if the difference is smaller than the predetermined threshold value, the inspection unit 78 determines the product as a "defective product."

[0133] Furthermore, the inspection unit 78 makes the above-mentioned judgment for all the test image data Ka to Kd relating to the inspection area of ​​the printed circuit board 1, and if the inspection unit 78 judges all of the test image data Ka to Kd to be "good", it judges the inspection area to be "good" and stores this result in the storage unit 57. On the other hand, if the inspection unit 78 makes the above-mentioned judgment for all of the test image data Ka to Kd relating to the inspection area and judges at least one of the test image data Ka to Kd to be "defective", it judges the inspection area to be "defective" and stores this result in the storage unit 57.

[0134] Then, if the above inspection process is performed on all inspected areas of the printed circuit board 1 and all inspected areas are judged to be "good," the solder fillet inspection device 16 judges that the printed circuit board 1 has no abnormalities in the solder fillets 5 (pass judgment) and stores this result in the memory unit 57.

[0135] On the other hand, if there is even one inspected area that is judged to be "defective," the solder fillet inspection device 16 judges that the printed circuit board 1 has an abnormality in the solder fillet 5 (fails the inspection), stores this result in the memory unit 57, and notifies the outside world of this via the display unit 56, communication unit 58, etc.

[0136] As described above in detail, according to this embodiment, the inspection image data Ka-Kd are each formed by providing one solder fillet image Ih in the image frames W1 and W2. Therefore, the size (width and height) of each of the inspection image data Ka-Kd is constant and does not vary depending on the size of the land 3. This eliminates the need to prepare multiple AI models (identification means) that differ for each land 3 size, thereby reducing the effort and time required to obtain the AI ​​models 101 and 102. Furthermore, the AI ​​models 101 and 102 can be commonly used even when the land 3 sizes are different.

[0137] Furthermore, the image frames W1 and W2 of the learning data Ga-Gd and the image frames W1 and W2 of the test image data Ka-Kd are the same size, and the sizes of the learning data Ga-Gd and the test image data Ka-Kd are the same. Therefore, when the test image data Ka-Kd are input to the AI ​​models 101 and 102, appropriate reconstructed image data S corresponding to the test image data Ka-Kd can be more reliably output, and the quality of the solder fillet 5 can be more accurately determined. This makes it possible to more reliably obtain good test accuracy.

[0138] Additionally, the inspection image data Ka-Kd are compared with the reconstructed image data S, which is reconstructed by inputting the inspection image data Ka-Kd into the AI ​​models 101 and 102, and the quality of the solder fillet 5 is determined based on the comparison results. Therefore, the image data Ka-Kd and S to be compared each relate to the same solder fillet 5. Therefore, unlike a method of determining quality by comparison with a separately prepared standard, it is not necessary to set relatively loose inspection conditions to prevent false detection, and stricter inspection conditions can be set. Furthermore, the imaging conditions of the printed circuit board 1 to be inspected (e.g., the position, angle, and deflection of the printed circuit board 1) and the imaging conditions of the inspection device 16 (e.g., lighting conditions, camera angle, etc.) can be matched between the image data Ka-Kd and S to be compared. This combination allows for more accurate quality determination of the solder fillet 5.

[0139] Furthermore, in this embodiment, one solder fillet image Ih constituting the inspection image data Ka to Kd includes not only an image of the portion of the solder fillet 5 located on the land 3, but also an image of the portion of the solder fillet 5 that protrudes from the land 3. This makes it possible to properly determine whether the solder fillet 5 that has a portion protruding from the land 3 is good or bad, further improving inspection accuracy.

[0140] Furthermore, in this embodiment, when the size of one solder fillet image Ih is relatively small, inspection image data Kc, Kd of relatively small size are obtained by providing one solder fillet image Ih in a relatively small second image frame W2. These relatively small inspection image data Kc, Kd are then input to the second AI model 102, which outputs reconstructed image data S, and the relatively small inspection image data Kc, Kd and the reconstructed image data S are compared. Therefore, compared to when the image frame of the inspection image data is always set to a fixed size, the process for obtaining the reconstructed image data S and the comparison process by the inspection unit 78 can be speeded up, and the inspection speed can be further improved.

[0141] In addition, because the orientation and position of the solder fillet 5 in the learning data Ga to Gd and the inspection image data Ka to Kd are roughly aligned, it is possible to accurately determine whether the solder fillet 5 is good or bad even if a relatively small amount of learning data Ga to Gd is used to generate the AI ​​models 101 and 102. In other words, it is possible to obtain good inspection accuracy while more effectively reducing the effort and time required to obtain the AI ​​models 101 and 102.

[0142] The present invention is not limited to the above-described embodiment, and may be implemented as follows: Of course, other applications and modifications not exemplified below are also possible.

[0143] (a) In the above embodiment, the entire test image data Ka to Kd is compared with the entire reconstructed image data S in the pass / fail determination process of step S304.

[0144] In contrast to this, it is also possible to configure the comparison so that only one solder fillet image Ih in the inspection image data Ka to Kd is used as the comparison target, and to compare the inspection image data Ka to Kd with the reconstructed image data S. In other words, it is also possible to configure the comparison so that one solder fillet image Ih in the inspection image data Ka to Kd is compared with an area of ​​the reconstructed image data S that overlaps with the one solder fillet image Ih.

[0145] In this configuration, since the portions of the inspection image data Ka-Kd other than the one solder fillet image Ih are not compared, the processing load associated with comparing the two image data Ka-Kd, S can be reduced compared to comparing the entirety of both image data Ka-Kd, S. This allows for faster and more efficient inspection. Furthermore, it is possible to more reliably prevent portions of the inspection image data Ka-Kd other than the one solder fillet image Ih, i.e., portions unrelated to the solder fillet 5, from affecting the pass / fail judgment, thereby further improving inspection accuracy.

[0146] It is also possible to configure the comparison between the inspection image data Ka-Kd and the reconstructed image data S by comparing only the area relating to the solder fillet 5 in the reconstructed image data S. In other words, it is also possible to configure the comparison to compare the area relating to the solder fillet 5 in the reconstructed image data S with the area in the inspection image data Ka-Kd that overlaps with the area relating to the solder fillet 5. Of course, the above two comparison methods may be used in combination.

[0147] (b) In the above embodiment, the learning data Ga to Gd are obtained using the original learning image data Ig of the printed circuit board 1 that passed the post-reflow inspection in training the neural network 90. ​​In contrast to this, the learning data Ga to Gd may be obtained using original learning image data of non-defective solder fillets 5 that have been visually selected by an operator after the reflow process, for example.

[0148] The learning unit 77 may also acquire the learning data Ga to Gd using image data of a virtually generated non-defective solder fillet 5 .

[0149] (c) In the above embodiment, separate AI models 101 and 102 are provided, one corresponding to two-dimensional data and one corresponding to three-dimensional data, but it is also possible to provide a common AI model that corresponds to both two-dimensional data and three-dimensional data.

[0150] Furthermore, the second AI model 102 may be omitted. In this case, the size of the second test image data Kc, Kd may be set to be the same as the size of the first test image data Ka, Kd, ​​and the first AI model 101 may be configured to perform testing based on the test image data Ka to Kd.

[0151] (d) The configuration of the AI ​​models 101, 102 (neural network 90) and the learning method thereof are not limited to those described in the above embodiment. For example, the neural network 90 may be configured to perform normalization or other processing on various data as needed during the learning process or the reconstructed image data acquisition process. Furthermore, the structure of the neural network 90 is not limited to that shown in FIG. 5 , and may include, for example, a pooling layer after the convolution layer 93. Of course, the number of layers of the neural network 90, the number of nodes in each layer, and the connection structure of each node may be different.

[0152] Furthermore, in the above embodiment, the AI ​​models 101 and 102 (neural network 90) are generative models having the structure of a convolutional autoencoder (CAE), but this is not limited to this, and they may also be generative models having the structure of a different type of autoencoder, such as a variational autoencoder (VAE).

[0153] Furthermore, in the above embodiment, the neural network 90 is configured to learn using the error backpropagation method, but this is not limiting, and the neural network 90 may be configured to learn using various other learning algorithms.

[0154] Additionally, neural network 90 may be configured by a dedicated AI processing circuit such as an AI chip. In this case, only learning information such as parameters may be stored in storage unit 57, and the dedicated AI processing circuit may read this information and set it in neural network 90, thereby configuring AI models 101 and 102.

[0155] Additionally, in the above embodiment, the control device 33 is provided with the learning unit 77, and the neural network 90 is configured to be learned within the control device 33, but this is not limiting. For example, the learning unit 77 may be omitted, and the neural network 90 may be configured to be learned outside the control device 33, and the AI ​​models 101 and 102 (trained neural network 90) that have been trained outside may be stored in the storage unit 57.

[0156] (e) In the above embodiment, two-dimensional data and three-dimensional data are acquired as the original inspection image data Ik, but it is also possible to acquire only one of the two-dimensional data and the three-dimensional data. Furthermore, the AI ​​models 101 and 102 may be provided to correspond to only one of the two-dimensional data and the three-dimensional data, depending on the data to be acquired.

[0157] (f) In the above embodiment, the inspection image data Ka-Kd are obtained by pasting one solder fillet image Ih into the image frames W1 and W2. Alternatively, the inspection image data Ka-Kd may be obtained as follows. That is, first, an image of the connected components (lump portion) in the area occupied by the identified solder fillet 5 is extracted as one solder fillet image Ih, and then the one solder fillet image Ih and its surrounding area are extracted to obtain an extracted image of the same size as the image frames W1 and W2. The inspection image data Ka-Kd may then be obtained by replacing the values ​​of each pixel in the surrounding area of ​​the extracted image with the same value (for example, by setting brightness or height to "0"). Of course, the learning data Ga-Gd may also be obtained using a similar method.

[0158] 1...Printed circuit board, 3...Land, 5...Solder fillet, 6...Electronic component, 16...Solder fillet inspection device, 32d...Camera (image data acquisition means), 78...Inspection unit (inspection image data acquisition means, reconstructed image data acquisition means, comparison means, solder fillet image extraction means), 90...Neural network, 91...Encoder unit (encoding unit), 92...Decoder unit (decoding unit), 101...First AI model (identification means), 102...Second AI model (second identification means) Ih...Solder fillet image of 1, W1...First image frame (image frame), W2...Second image frame.

Claims

1. A solder fillet inspection device for inspecting solder fillets that solder electronic components on a printed circuit board, comprising: image data acquisition means capable of acquiring image data of a predetermined inspection area on the printed circuit board including the solder fillet; discrimination means for generating image data by having a neural network having an encoding unit that extracts features from input image data and a decoding unit that reconstructs image data from the features learn only image data related to good solder fillets as learning data; inspection image data acquisition means for acquiring inspection image data including an image of the solder fillet to be inspected based on the image data acquired by the image data acquisition means; reconstructed image data acquisition means for inputting the inspection image data to the discrimination means and acquiring reconstructed image data as reconstructed image data; and comparison means for comparing the inspection image data and the reconstructed image data, and the device is configured to be able to determine whether the solder fillet is good or bad based on the comparison result by the comparison means, and the learning data is composed of one solder fillet image showing a solder fillet corresponding to one land, arranged in an image frame larger than the size of the one solder fillet image, The solder fillet inspection device is characterized in that the inspection image data acquisition means acquires the inspection image data of the same size as the learning data, wherein the one solder fillet image extracted from the image data acquired by the image data acquisition means is placed in an image frame of the same size as the image frame of the learning data.

2. A solder fillet inspection device as described in claim 1, further comprising a solder fillet image extraction means for extracting the one solder fillet image constituting the inspection image data from the image data acquired by the image data acquisition means, wherein the solder fillet image extraction means is capable of identifying an area occupied by a solder fillet in the image data acquired by the image data acquisition means, and extracting an image of a connected component in the identified area as the one solder fillet image constituting the inspection image data.

3. A solder fillet inspection device as described in claim 1, further comprising a second identification means for generating second learning data by having a neural network having an encoding unit for extracting features from input image data and a decoding unit for reconstructing image data from the features, the second learning data being generated by having the first solder fillet image be trained in a second image frame that is larger than the size of the first solder fillet image but smaller than the size of the image frame of the learning data, and when the size of the first solder fillet image extracted from the image data acquired by the image data acquisition means is smaller than the size of the second image frame, the inspection image data acquisition means acquires the inspection image data of the same size as the second learning data in which the first solder fillet image is arranged in the second image frame, the reconstructed image data acquisition means acquires the reconstructed image data reconstructed by inputting the inspection image data to the second identification means, and the comparison means is configured to compare the inspection image data and the reconstructed image data.

4. The solder fillet inspection device of claim 1, characterized in that the learning data and the inspection image data are set so that the center or center of gravity of the one solder fillet image coincides with the center of the image frame and the part of the one solder fillet image on the electronic component side faces a predetermined direction.

5. The solder fillet inspection device as described in claim 1, characterized in that the comparison means is configured to be able to compare the inspection image data and the reconstructed image data by using only the one solder fillet image in the inspection image data as a comparison object.

6. A solder fillet inspection method for inspecting solder fillets that solder electronic components on a printed circuit board, comprising: an image data acquisition step capable of acquiring image data of a predetermined inspection area on the printed circuit board including the solder fillet; an inspection image data acquisition step acquiring inspection image data including an image of the solder fillet to be inspected based on the image data acquired by the image data acquisition step; a reconstructed image data acquisition step using a discrimination means generated by having a neural network having an encoding unit that extracts features from input image data and a decoding unit that reconstructs image data from the features learn only image data related to good solder fillets as learning data, and inputting the inspection image data acquired by the inspection image data acquisition step into the discrimination means to acquire reconstructed image data as reconstructed image data; and a comparison step comparing the inspection image data and the reconstructed image data, and judging whether the solder fillet is good or bad based on the comparison result in the comparison step, wherein the learning data is one solder fillet image showing a solder fillet corresponding to one land, and is provided in an image frame larger than the size of the one solder fillet image, A solder fillet inspection method characterized in that, in the inspection image data acquisition process, the one solder fillet image extracted from the image data acquired by the image data acquisition process is arranged in an image frame of the same size as the image frame of the learning data, and the inspection image data of the same size as the learning data is acquired.

Images