Information processing device and information processing method

Abstract

To provide an information processing device capable of accurately determining appropriateness about a position at which a component is mounted on a board before reflow in manufacturing of a printed board by soldering.SOLUTION: According to an embodiment, an information processing device includes an image processing part and a determination part. The image processing part generates a composite image obtained by compositing a component in a pre-mounting image on the basis of the pre-mounting image before mounting the component on a board, absorption information showing an absorption state of the component in a component mounting device, and setting information showing a setting position set as a position at which the component is mounted on the board. The determination part inputs image data based on the pre-mounting image and the composite image in a machine learning model for outputting an inspection result in inspection after reflow from image data based on an image before the reflow, and determines appropriateness of mounting the component at the setting position.SELECTED DRAWING: Figure 4

Description

Technical Field

[0001] Embodiments of the present invention relate to an information processing apparatus and an information processing method.

Background Art

[0002] When joining components to a substrate such as a printed wiring board or a printed circuit board by soldering, the components are mounted on the substrate in the order of normal-temperature solder and components. Then, in a state where the solder and the components are mounted on the substrate, reflow is performed, so that the solder is melted and the components are joined to the substrate by soldering. In a component mounting apparatus (mounter), a nozzle sucks a component to be mounted on a substrate from a parts feeder or a tray, and mounts the component adsorbed on the nozzle on the substrate on which the solder is mounted.

[0003] When manufacturing a printed circuit board by soldering as described above, from the viewpoints of suppressing an increase in member loss due to the occurrence of defective products and an increase in the number of processes due to the repair of defective products, detection of defects in the printed circuit board before reflow is performed. For example, an image showing the adsorption state of a component on a nozzle in a component mounting apparatus is acquired by an imaging device such as a camera, and based on the adsorption state of the component shown in the image, it is determined whether the component is properly joined to the substrate. When determining whether a component is properly joined to a substrate before reflow as described above, in addition to the adsorption state of the component in the component mounting apparatus, it is required that the determination be made in consideration of the state of the substrate before mounting the component, such as the mounting state of the solder on the substrate. That is, it is required to accurately determine the appropriateness of the position where the component is mounted on the substrate before reflow, considering the adsorption state of the component in the component mounting apparatus and the state of the substrate before mounting the component.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

[0005] The problem that this invention aims to solve is to provide an information processing device and an information processing method that enable accurate determination of the appropriateness of the position of components to be mounted on a printed circuit board before reflow soldering in the manufacturing of printed circuit boards by soldering. [Means for solving the problem]

[0006] According to the embodiment, an information processing device for soldering components to a substrate is provided. The information processing device comprises an image processing unit and a determination unit. The image processing unit generates a composite image by compositing a component onto the pre-mounting image based on a pre-mounting image before mounting a component onto the substrate, adsorption information indicating the adsorption state of the component in the component mounting device, and setting information indicating a set position set as the position where the component will be mounted on the substrate. The determination unit determines the appropriateness of mounting the component onto the substrate at the set position set in the setting information by inputting the pre-mounting image and the composite image image into a machine learning model that outputs inspection results in the inspection after reflow from the input of image data based on the image before reflow. [Brief explanation of the drawing]

[0007] [Figure 1] Figure 1 is a schematic diagram showing a manufacturing system according to the first embodiment as an example of a manufacturing system for producing printed circuit boards. [Figure 2] Figure 2 is a schematic diagram showing a component mounting device and its vicinity in a manufacturing system according to the first embodiment. [Figure 3] Figure 3 is a block diagram schematically showing the configuration of the information processing device of the manufacturing system according to the first embodiment. [Figure 4] Figure 4 is a schematic diagram showing an example of processing in the image processing unit, determination unit, and position search unit of the information processing device of the manufacturing system according to the first embodiment. [Figure 5] Figure 5 is a schematic diagram showing an example of generating a composite image based on a pre-mounting image, adsorption information, and setting information in an information processing device according to the first embodiment. [Figure 6] Figure 6 is a schematic diagram showing an example of a machine learning model used for determination by the determination unit 32 in the information processing device according to the first embodiment. [Figure 7] Figure 7 is a flowchart illustrating an example of the machine learning model generation process (construction process) by the learning model generation unit in the information processing apparatus according to the first embodiment. [Figure 8] Figure 8 is a schematic diagram showing an example of processing in the image processing unit, determination unit, and position search unit of an information processing device for a manufacturing system according to a certain modification. [Modes for carrying out the invention]

[0008] The embodiments and other details will be described below with reference to the drawings.

[0009] (First embodiment) First, as an example of an embodiment, the first embodiment will be described. Figure 1 shows a manufacturing system 1 according to the first embodiment as an example of a manufacturing system for manufacturing printed circuit boards. As shown in Figure 1, the manufacturing system 1 includes a solder printing device 2, a component mounting device (mounter) 3, a reflow device 4, an inspection device 5, and an information processing device 6. In the manufacturing system 1, a printed circuit board manufacturing line is formed, and in the manufacturing line, the solder printing device 2, component mounting device 3, reflow device 4, and inspection device 5 are arranged in that order from the upstream side.

[0010] In the manufacturing line, the substrate is fed into the solder printing device 2. The substrate may be a printed wiring board, or a printed circuit board (printed circuit board) on which components such as chip components are already attached. The solder printing device 2 applies solder to the substrate by, for example, printing solder onto pads or lands formed on the surface of the substrate. In one example, solder may be applied to the substrate by printing solder onto the surface of components already attached to the printed circuit board. In the solder printing device 2, room temperature solder is applied to the substrate, and the solder is applied to the substrate in an unmelted state. Note that the method of applying solder to the substrate is not limited to printing. In one example, solder may be applied to the substrate by dispensing (coating) solder or by attaching a solder sheet.

[0011] A circuit board with solder already applied is fed into the component mounting device 3. The component mounting device 3 mounts components to be newly attached to the circuit board. Examples of components mounted on the circuit board include chip components and IC packages. Once the components are mounted, the solder is sandwiched between the circuit board and the components. The circuit board with solder and components mounted is fed into the reflow device 4. The reflow device 4 performs reflow soldering. Reflow melts the solder already on the circuit board, and the newly mounted components are joined to the circuit board. This forms a printed circuit board with the mounted components attached to the circuit board. In one example, the newly mounted components are joined by soldering to pads or lands formed on the surface of the circuit board. In another example, the newly mounted components are joined by soldering to components already mounted on the circuit board.

[0012] The inspection device 5 receives manufactured printed circuit boards, that is, printed circuit boards in which components have been bonded to the board by reflow in the reflow machine 4. The inspection device 5 inspects the manufactured printed circuit boards and determines whether they are good or bad. Only printed circuit boards that are determined to be good by the inspection device 5 are released to the market as products. For example, the inspection device 5 performs visual inspection and electrical testing on the manufactured printed circuit boards. In the visual inspection, an image of the printed circuit board in which components have been bonded to the board by reflow is acquired by taking a photograph or other means, and the quality of the manufactured printed circuit board is determined based on the acquired image of the printed circuit board. In the electrical test, electricity is passed through the manufactured printed circuit board, and the amount of electricity is measured by a tester to determine whether the printed circuit board is good or bad. The inspection device 5 transmits information indicating the inspection results of the actual inspections performed to the information processing device 6.

[0013] Furthermore, the manufacturing system 1 is equipped with imaging devices 11 to 13. Each of the imaging devices 11 to 13 is, for example, a camera or a video camera. Imaging device 11 is positioned upstream of the solder printing device 2 and captures an image of the substrate only. Imaging device 12 is positioned between the solder printing device 2 and the component mounting device 3 and captures an image of the substrate with only solder mounted on it. As a result, in the manufacturing system 1, imaging devices 11 and 12 capture two types of images: an image of the substrate only and an image of the substrate with only solder mounted on it, as images before reflow by the reflow device 4. In addition, the image of the substrate only and the image of the substrate with only solder mounted on it are captured as pre-mounting images before components are mounted on the substrate by the component mounting device 3.

[0014] Figure 2 shows the component mounting device 3 and its vicinity. In Figure 2, the X direction (indicated by arrow X) coincides with or approximately coincides with the transport direction in the substrate manufacturing line. The Y direction (perpendicular or approximately perpendicular to the plane of Figure 2) coincides with or approximately coincides with the width direction in the substrate manufacturing line and is perpendicular or approximately perpendicular to the transport direction in the manufacturing line. As shown in Figures 1 and 2, the imaging device 13 is positioned in or near the component mounting device 3. The component mounting device 3 is also equipped with a nozzle 15. In the component mounting device (mounter) 3, the nozzle 15 picks up components 16 to be mounted on the substrate from a parts feeder or tray. The components 16 picked up by the nozzle 15 are then mounted on the substrate, which already has solder applied.

[0015] The imaging device 13 captures images of the component 16 being held in place by the nozzle 15 of the component mounting device 3. The imaging device 13 captures images of the nozzle 15 and the component 16 held in place by the nozzle 15 from one side in a direction that intersects (orthogonal or nearly orthogonal to) both the X and Y directions. In one example, if the substrate manufacturing line extends horizontally or nearly horizontally, the imaging device 13 captures images of the nozzle 15 and the component 16 held in place by the nozzle 15 from the vertically downward side. As described above, the imaging device 13 captures images of the component 16 held in place by the nozzle 15 as suction information indicating the suction state of the component 16 in the component mounting device 3.

[0016] Each of the imaging devices 11 to 13 transmits the captured image (image data) to the information processing device 6. Therefore, the information processing device 6 acquires the two types of images captured by imaging devices 11 and 12 as pre-reflow images and pre-mounting images before components are mounted. The information processing device 6 also acquires the image captured by imaging device 13 as suction information indicating the suction state of components in the component mounting device 3. Furthermore, each of the imaging devices 11 to 13 takes images in real time each time a substrate is brought into the manufacturing line. Therefore, the information processing device 6 acquires, in real time, the pre-mounting images captured by imaging devices 11 and 12 before components are mounted, and the image captured by imaging device 13 as suction information indicating the suction state, each time a substrate is brought into the manufacturing line.

[0017] Further, in the present embodiment, the information processing apparatus 6 transmits a command regarding mounting of components on a substrate to the component mounting apparatus 3. In one example, in the command from the information processing apparatus 6 to the component mounting apparatus 3, that is, in the command regarding mounting of components on a substrate, the X coordinate in the X direction (transport direction) and the Y coordinate in the Y direction (width direction of the manufacturing line) regarding the position where the component is to be mounted on the substrate are shown as position information. Also, when the posture of the component to be mounted on the substrate can be adjusted in the component mounting apparatus 3, in the command regarding mounting of components on the substrate, in addition to the X coordinate and the Y coordinate serving as position information, posture information regarding the posture of mounting the component on the substrate may be shown. In this case, in the component to be mounted on the substrate, a reference direction such as the axial direction or the longitudinal direction is defined. Then, the angle formed by the reference direction in the component with respect to the X direction or the Y direction is shown as the posture information.

[0018] The component mounting apparatus 3 mounts the component on the substrate at a position corresponding to the command from the information processing apparatus 6. Also, when the above-described posture information is shown in the command regarding mounting of components on the substrate, the component mounting apparatus 3 mounts the component on the substrate in the posture corresponding to the command from the information processing apparatus 6.

[0019] FIG. 3 shows an example of the configuration of the information processing apparatus 6 according to the present embodiment. As shown in FIG. 3, the information processing apparatus 6 includes a processing execution unit 21, a storage unit 22, a communication interface 23, and a user interface 25. The processing execution unit 21 includes an image processing unit 31, a determination unit 32, a position search unit 33, a learning model generation unit 35, and a learning model update unit 36. Each of the image processing unit 31, the determination unit 32, the position search unit 33, the learning model generation unit 35, and the learning model update unit 36 performs a part of the processing performed by the processing execution unit 21.

[0020] In one example, the information processing device 6 is composed of a computer or the like, and the computer's processor or integrated circuit functions as the processing execution unit 21. The computer's processor or integrated circuit includes any of the following: CPU (Central Processing Unit), ASIC (Application Specific Integrated Circuit), GPU (Graphics Processing Unit), microcontroller, FPGA (Field Programmable Gate Array), and DSP (Digital Signal Processor). The computer that becomes the information processing device 6 may have only one integrated circuit or the like, or it may have multiple integrated circuits or the like.

[0021] Furthermore, in the computer that serves as the information processing device 6, the computer's storage medium functions as the storage unit 22. The storage medium may include not only main memory such as memory, but also auxiliary storage devices. Examples of storage media include magnetic disks, optical disks (CD-ROM, CD-R, DVD, etc.), magneto-optical disks (MO, etc.), and semiconductor memory. The computer that serves as the information processing device 6 may be equipped with only one storage medium, or it may be equipped with multiple storage media.

[0022] In the computer that constitutes the information processing device 6, the processor or integrated circuit executes a program stored in a storage medium, etc., thereby performing the processing described later by the processing execution unit 21. In one example, the program executed by the processor, etc., in the computer that constitutes the information processing device 6 may be stored in a computer (server) connected via a network such as the Internet, or in a server in a cloud environment. In this case, the processor downloads the program via the network. In another example, the information processing device 6 is composed of multiple computers that are separate from each other. In this case, the processing described later by the processing execution unit 21 is performed by the processors or integrated circuits, etc., of the multiple computers.

[0023] In one example, the information processing device 6 is comprised of a server in a cloud environment. The infrastructure of the cloud environment consists of virtual processors such as virtual CPUs and cloud memory. In the cloud environment server that becomes the information processing device 6, the virtual processors function as the processing execution unit 21, and the processing described later is performed by the processing execution unit 21. The cloud memory functions as the storage unit 22.

[0024] In the information processing device 6, the communication interface 23 consists of interfaces for accessing external devices such as the component mounting device 3, the inspection device 5, and the imaging devices 11-13. The information processing device 6 can communicate with external devices via the communication interface 23, either by wire or wirelessly. Therefore, the information processing device 6 acquires information indicating the inspection results from the inspection device 5, and images captured by the imaging devices 11-13, etc., via the communication interface 23. The information processing device 6 also transmits commands related to component mounting to the component mounting device 3 via the communication interface 23.

[0025] Furthermore, in the information processing device 6, various operations are input by users of the manufacturing system 1 via the user interface 25. The user interface 25 is equipped with buttons, switches, and touch panels as operating elements into which users of the manufacturing system 1 can input operations. In addition, various information is communicated to users of the manufacturing system 1 via the user interface 25. This communication is carried out, for example, by screen display and audio transmission. In one example, the user interface 25 is provided as an external device of the information processing device 6 and is separate from the computer and other components that make up the information processing device 6.

[0026] The following describes the processing performed by the processing execution unit 21. Figure 4 shows an example of the processing performed by the image processing unit 31, the determination unit 32, and the position search unit 33. The processing shown in Figure 4 is performed each time a substrate is brought into the manufacturing line. As shown in Figure 4, the position search unit 33 sets a designated position as the position where components will be mounted on the substrate (S101). The position search unit 33 then generates setting information indicating the designated position set as the position where components will be mounted on the substrate. The image processing unit 31 acquires an image I1 of only the substrate from the imaging device 11 and an image I2 of the substrate with only solder mounted from the imaging device 12 as pre-mounting images before components are mounted. The image processing unit 31 also acquires an image taken by the imaging device 13 as suction information indicating the suction state of the components in the component mounting device 3. The image processing unit 31 then acquires setting information indicating the designated position where components will be mounted on the substrate.

[0027] The image processing unit 31 generates a composite image I3 by combining (superimposing) the image of the component onto image I2, which is the image before mounting, based on image I2, the adsorption information, and the setting information (S102). In generating the composite image I3, the image processing unit 31 extracts the position and orientation of the component in the adsorption state indicated by the adsorption information, based on the adsorption information. Then, the image processing unit 31 calculates the position and orientation of the component in image I2 corresponding to the adsorption state in the adsorption information, based on the position and orientation of the component extracted from the adsorption information. At this time, for example, the position and orientation of the component in image I2 corresponding to the adsorption state in the adsorption information is calculated using a relational expression that transforms the coordinate system of the image that will become the adsorption information to the coordinate system of image I2. In this case, data showing the relationship between the coordinate system of the image that will become the adsorption information and the coordinate system of image I2, such as a relational expression that transforms the coordinate system of the image that will become the adsorption information to the coordinate system of image I2, is stored in the storage unit 22, etc. In one example, the image processing unit 31 calculates the X-coordinate in the X direction and the Y-coordinate in the Y direction as the position of the part in image I2 corresponding to the adsorption state in the adsorption information. The image processing unit 31 also calculates the angle that the reference direction of the part makes with respect to the X direction or the Y direction as the orientation of the part in image I2 corresponding to the adsorption state in the adsorption information.

[0028] The image processing unit 31 then determines the position and orientation of the component to be superimposed in the composite image I3 based on the position and orientation of the component in image I2 corresponding to the adsorption state in the adsorption information, and the setting position set in the setting information for the position where the component is mounted. In one example, the setting information sets the displacement of the component relative to the position in image I2 corresponding to the adsorption state in the adsorption information as the setting position. For example, suppose (X,Y)=(Xa,Ya) is calculated as the position of the component in image I2 corresponding to the adsorption state in the adsorption information. Also, suppose the orientation of the component in image I2 corresponding to the adsorption state in the adsorption information is indicated by the angle θ that the reference direction of the component makes with respect to the Y direction, and the angle θa is calculated as the value of angle θ. Then, suppose the setting information indicates the setting position as the displacement Xb in the X direction and the displacement Yb in the Y direction from the coordinate (Xa,Ya), and (Xb,Yb)=(Xb0,Yb0) is set. In this case, the component is superimposed in composite image I3 (image I2) at a position where (X,Y) = (Xa + Xb0, Ya + Yb0). Then, the component is composited in composite image I3 with the angle θ that the reference direction of the component makes with respect to the Y direction being angle θa.

[0029] In one example, the component mounting device 3 allows for adjustment of the orientation of the component mounted on the substrate, and the orientation of the component mounted on the substrate can be changed. In this case, the setting information includes the aforementioned setting position, and the displacement of the component's orientation in image I2, corresponding to the adsorption state in the adsorption information, is set as the setting orientation. For example, in the setting information, the setting orientation is indicated by the displacement θb from the aforementioned angle θa with respect to the angle θ that the reference direction of the component makes with respect to the Y direction, and it is set to θb = θb0. In this case, the component is composited in composite image I3 when the angle θ that the reference direction of the component makes with respect to the Y direction is angle θa + θb0.

[0030] When the composite image I3 is generated, the image processing unit 31 generates image data to be used in the processing of the determination unit 32 as pre-processing (S103). The image processing unit 31 generates image data by performing image processing using two pre-mounting images, I1 and I2, and the composite image I3, before the components are mounted. As a result, image data based on images I1, I2 and the composite image I3 is generated as image data to be used in the processing of the determination unit 32.

[0031] In one example, the image processing unit 31 converts images I1, I2, and composite image I3 to grayscale. The image processing unit 31 then aligns images I1, I2, and composite image I3 relative to each other, correcting any positional misalignment between the three images I1 to I3. The image processing unit 31 then assigns different color information to the three grayscale images I1 to I3. For example, image I1 is assigned blue color information, image I2 is assigned green color information, and composite image I3 is assigned red color information. The image processing unit 31 then combines the three images I1 to I3, each with different color information. As a result, the combined image data of the three images I1 to I3 is generated and used as image data for processing in the determination unit 32. Therefore, in this example, after aligning the three types of images I1 to I3 and assigning different color information to each of the three types of images I1 to I3, the three types of images I1 to I3 are combined to generate image data used for processing in the determination unit 32.

[0032] In another example, the image processing unit 31 arranges images I1 and I2, which are pre-mounting images of the component before it is mounted, and a composite image I3. As a result, image data consisting of the three types of images I1 to I3 arranged together is generated as image data used for processing in the determination unit 32. Therefore, in this example, the image data used for processing in the determination unit 32 consists of the three types of images I1 to I3 arranged relative to each other.

[0033] The memory unit 22 stores the machine learning model generated (constructed) as described later. As shown in Figure 4, the determination unit 32 uses the machine learning model to determine whether the inspection to be performed by the inspection device 5 after reflow is good or bad from the image data generated by the image processing unit 31 in S103 (S104). That is, the determination unit 32 determines whether the inspection to be performed after reflow is good or bad from the image data based on images I1 and I2 acquired in real time before the components are mounted, and a composite image I3 obtained by combining the components with image I2. The determination unit 32 inputs the image data based on images I1 to I3 generated by the image processing unit 31 into the machine learning model. The determination unit 32 then has the machine learning model output whether the inspection is good or bad as the inspection result, and the inspection result output from the machine learning model is used as the determination result regarding whether the inspection is good or bad.

[0034] Furthermore, the determination unit 32 determines, based on the inspection results output from the machine learning model, whether it is appropriate to mount the components on the circuit board at the set positions specified in the configuration information. In this case, if the inspection results output from the machine learning model are good, the determination unit 32 determines that it is appropriate to mount the components on the circuit board at the set positions. On the other hand, if the inspection results output from the machine learning model are bad, the determination unit 32 determines that it is not appropriate to mount the components on the circuit board at the set positions.

[0035] If the inspection result output from the machine learning model is good, that is, if it is appropriate to mount the component on the substrate at the set position, the position search unit 33 generates a command to mount the component on the substrate at the set position set in the setting information as a command regarding the mounting of the component on the substrate (S105). The position search unit 33 then transmits the generated command to the component mounting device 3, and the component mounting device 3 mounts the component on the substrate at the set position set in the setting information. For example, suppose that in the composite image I3, as described above, a component is superimposed on image I2 at the position where (X,Y)=(Xa+Xb0,Ya+Yb0). Suppose that the machine learning model outputs a good inspection result in response to the input image data based on images I1 to I3. In this case, the position search unit 33 causes the component mounting device 3 to mount the component on the substrate at the position corresponding to (X,Y)=(Xa+Xb0,Ya+Yb0). In other words, the position search unit 33 causes the component mounting device 3 to mount the component onto the substrate at the set position (Xb0, Yb0).

[0036] Furthermore, if the inspection result output from the machine learning model is defective, that is, if it is not appropriate to mount the circuit board on the component at the set position, the position search unit 33 changes the aforementioned set position set in the setting information (S101). In other words, the set position set in the setting information as the position for mounting the component on the circuit board is reset, and the setting information is updated. For example, suppose in the composite image I3, as described above, the component is superimposed on image I2 at the position (X,Y)=(Xa+Xb0,Ya+Yb0). Then, suppose the machine learning model outputs a defective inspection result in response to the input of image data based on images I1 to I3. In this case, for example, the set position indicated in the setting information is changed from (Xb,Yb)=(Xb0,Yb0) to (Xb,Yb)=(Xb1,Yb1).

[0037] Then, the image processing unit 31 generates a new composite image I3 by combining the component with image I2, which is the image before mounting, the adsorption information, and the setting information with the changed setting position (S102). At this time, the position of the component to be superimposed in the composite image I3 is determined based on the setting information with the changed setting position. For example, if the setting position indicated in the setting information is changed to (Xb,Yb)=(Xb1,Yb1), the component will be superimposed in the newly generated composite image I3 at the position where (X,Y)=(Xa+Xb1,Ya+Yb1). As described above, the image processing unit 31 updates the composite image I3 by generating a new composite image I3. When generating a new composite image I3, the setting position in the setting information is changed from the previous generation of composite image I3, but the same data as the previous composite image I3 is used for image I2 and the image that will be used as the adsorption information.

[0038] Figure 5 shows an example of generating a composite image I3 based on image I2, adsorption information, and setting information. In the example in Figure 5, image I2 shows a state where only solder 42A and 42B are mounted on the substrate 41. Then, in the example in Figure 5, composite image I3 is generated by compositing component 43 onto image I2 based on the adsorption information and setting information. If the setting position is set to (Xb,Yb)=(Xb0,Yb0) in the setting information, component 43 is composited onto image I2 at the position where (X,Y)=(Xa+Xb0,Ya+Yb0), and composite image I3α is generated as composite image I3. Also, if the setting position is set to (Xb,Yb)=(Xb1,Yb1) in the setting information, component 43 is composited onto image I2 at the position where (X,Y)=(Xa+Xb1,Ya+Yb1), and composite image I3β is generated as composite image I3.

[0039] In the example shown in Figure 5, the setting positions used to generate the composite images I3α and I3β are different relative to each other. Therefore, in the composite images I3α and I3β, the composite component 43 is shifted relative to each other in both the X direction (left-right direction of composite images I3α and I3β) and the Y direction (up-down direction of composite images I3α and I3β). Note that in each of the images I2, I3α, and I3β shown in Figure 5, the direction indicated by arrow X is the X direction, and the direction indicated by arrow Y is the Y direction.

[0040] As shown in Figure 4, when the composite image I3 is updated, the image processing unit 31 generates new image data to be used for processing in the determination unit 32 using the pre-mounting images I1 and I2, and the updated composite image I3 (S103). This updates the image data input to the machine learning model. When generating new image data to be input to the machine learning model, the composite image I3 is updated from the previously generated image data, but the same data as the previously generated image data is used for images I1 and I2.

[0041] The determination unit 32 then inputs the updated image data into a machine learning model to determine whether the inspection to be performed by the inspection device 5 after reflow is good or bad (S104). Based on this, the determination unit 32 determines the appropriateness of mounting the component on the substrate at the changed setting position in the setting information. If it is appropriate to mount the component on the substrate at the changed setting position, the position search unit 33 generates a command to mount the component on the substrate at the changed setting position as a command for mounting the component on the substrate (S105), and transmits the generated command to the component mounting device 3. On the other hand, if it is not appropriate to mount the component on the substrate at the changed setting position, the position search unit 33 further changes the setting position in the setting information (S101). Then, the processing execution unit 21 and others sequentially perform the processing from S102 onwards using the setting information with the further changed setting position.

[0042] As the processing execution unit 21 performs the processing described above, each time the inspection result output from the machine learning model is found to be defective, the setting position set in the setting information as the position where the component will be mounted on the substrate is changed. Based on the setting information with the changed setting position, the image data based on the composite image I3 and images I1 to I3 is updated, and the updated image data is input to the machine learning model to determine whether the inspection to be performed by the inspection device 5 after reflow is good or bad. In other words, in this embodiment, the setting position in the setting information is changed until the inspection result output from the machine learning model is good, and the appropriateness of mounting the component on the substrate at the changed setting position is determined using the machine learning model.

[0043] Therefore, the position search unit 33, etc., searches for a position on the substrate where the inspection result output from the machine learning model will be good, based on the determination result from the determination unit 32 using the machine learning model. As a result, the position where the inspection result output from the machine learning model is good is searched for as the position to mount the component on the substrate. The position search unit 33 then sends a command regarding the mounting of the component on the substrate to the component mounting device 3, etc., so that the component mounting device 3 mounts the component on the substrate at the position searched for as the position where the inspection result output from the machine learning model is good.

[0044] In one example, the orientation of the components mounted on the substrate can be adjusted in the component mounting device 3, and the orientation is set in addition to the position in the setting information. If the inspection result output from the machine learning model is poor, the position search unit 33 may change the setting position set in the setting information, as well as the setting orientation. In this case, for example, the setting position indicated in the setting information is changed from (Xb,Yb)=(Xb0,Yb0) to (Xb,Yb)=(Xb1,Yb1), and the setting orientation indicated in the setting information is changed from θb=θb0 to θb=θb1. In this example, the image processing unit 31 updates the composite image I3 based on the setting information with the changed setting position and setting orientation, and uses the updated composite image I3 to generate new image data to be input to the machine learning model as described above.

[0045] Figure 6 shows an example of a machine learning model used for determination by the determination unit 32. As shown in Figure 6, the machine learning model consists of an input layer 51, an output layer 52, and an intermediate layer (hidden layer) 53 between the input layer 51 and the output layer 52. When the aforementioned image data is input to the machine learning model, pixel information such as RGB values ​​is input to the input layer 51 for each pixel of all the images that make up the image data. For example, in the aforementioned example where image data created by combining images I1 to I3 is generated as image data to be input to the machine learning model, pixel information is input to the input layer 51 for each pixel of the combined image. In another example where image data in which images I1 to I3 are arranged is generated as image data to be input to the machine learning model, pixel information is input to the input layer 51 for each pixel of all three types of arranged images I1 to I3.

[0046] In the example machine learning model shown in Figure 6, the input layer 51 consists of the same number of nodes as the total number of pixels in all the images that make up the input image data. In the example shown in Figure 6, the total number of pixels in all the images that make up the input image data is k, and pixel information A1 to Ak is input to the input layer 51. In the example machine learning model shown in Figure 6, the output layer 52 consists of nodes that output "good" in the inspection and nodes that output "bad" in the inspection.

[0047] Furthermore, the intermediate layer 53 is composed of convolutional layers, pooling layers, and fully connected layers. In the convolutional layers, feature parts are extracted from the image by filtering each pixel. In the pooling layers, the image is reduced in size while maintaining the feature parts of the image. The processing in the pooling layers absorbs the positional shift of feature parts in the image that constitutes the image data input to the machine learning model. In the intermediate layer 53, image features are recognized by processing in multiple convolutional layers and multiple pooling layers. In the fully connected layer, the image data from which feature parts have been recognized by the convolutional and pooling layers is converted into one-dimensional data. Through the processing described above, feature parts in the image are recognized in the intermediate layer 53 of the machine learning model, and the recognized feature parts are converted into information used to determine whether the inspection is good or bad.

[0048] In the machine learning model, the inspection results from the post-reflow inspection are output by inputting image data based on the image before reflow. The input image data is generated based on three types of images, for example, an image of the board only, an image of the board with only solder attached, and an image of the board with both solder and components attached. In the generation of image data input to the machine learning model for the determination by the determination unit 32, image I1 is used as the image of the board only, image I2 is used as the image of the board with only solder attached, and composite image I3 is used as the image of the board with both solder and components attached.

[0049] The machine learning model used for determination in the determination unit 32 is generated (constructed) by the learning model generation unit 35. In generating the machine learning model, training data consisting of a large number of datasets is used. Each dataset of the training data shows image data based on images before reflow for printed circuit boards that have been inspected in the past. For example, image data based on three types of images is shown: an image of the board only, an image of the board with only solder attached, and an image of the board with solder and components attached. The image data shown in each dataset is generated using images before reflow, in the same manner as in any of the examples described above. That is, the image data shown in each dataset is generated by combining multiple types of images before reflow, or by arranging multiple types of images before reflow.

[0050] Furthermore, each of the training data sets shows the inspection results from the actual inspection performed after reflow for printed circuit boards that have been inspected in the past. Each of the data sets may show the aforementioned image data and inspection results for printed circuit boards that have been inspected in the past in manufacturing system 1, or it may show the aforementioned image data and inspection results for printed circuit boards that have been inspected in the past in another manufacturing system with a similar configuration to manufacturing system 1. As the training data is structured as described above, in each of the many training data sets, the image data based on the image before reflow and the inspection results from the actual inspection performed after reflow are associated for printed circuit boards that have been inspected in the past. Note that the printed circuit boards inspected differ among the many data sets.

[0051] Figure 7 shows an example of the machine learning model generation process (construction process) by the learning model generation unit 35. When the process in Figure 7 is started, the learning model generation unit 35 classifies the numerous training data sets into training data sets and evaluation data sets (S111). In one example, the numerous data sets are classified so that the ratio of training data sets to evaluation data sets is 9. Then, the learning model generation unit 35 trains the model using deep learning with the training data sets of the learning model (S112). In this case, for example, a neural network is trained as the model. The model is also trained using supervised learning, where the inspection results shown in each of the training data sets are given as correct answers. Through deep learning, the model learns the features contained in the image data of the training data sets, for example, it learns the features in the image data when the inspection result is poor.

[0052] Once training using the training dataset is complete, the learning model generation unit 35 evaluates the trained model using the evaluation dataset (S113). In the model evaluation, image data based on images before reflow is input to the trained model for each of the evaluation datasets. Then, for each of the evaluation datasets, the output results from the trained model are compared with the test results from the actual tests performed. Finally, as an indicator for evaluating the trained model, the accuracy rate of the output results from the model relative to the test results from the actual tests is calculated.

[0053] Once the evaluation using the evaluation dataset is complete, the learning model generation unit 35 determines whether the accuracy calculated as an indicator for evaluating the trained model is above a standard level (S114). In one example, it is determined that the accuracy is above the standard level based on an accuracy of 90% or higher. If the accuracy is above the standard level (S114-Yes), the learning model generation unit 35 stores the trained model in the memory unit 22 as the machine learning model used for the determination by the determination unit 32 (S115).

[0054] If the accuracy rate is lower than the standard level (S114-No), the learning model generation unit 35 adds a dataset to the training data (S116). Then, the process returns to S111, and the learning model generation unit 35 sequentially performs the processes from S111 onward. This allows for the training of the model using the added dataset and the evaluation of the model using the added dataset. Note that the learning model generation unit 35 does not need to be located in the information processing device 6. In one example, the machine learning model generation process (construction process) described above may be performed by a computer or other device separate from the information processing device 6.

[0055] Furthermore, in the information processing device 6, the learning model update unit 36 ​​updates the machine learning model by relearning it. The learning model update unit 36 ​​stores the updated machine learning model in the memory unit 22. In one example, whenever the test result output from the machine learning model in the judgment of the judgment unit 32 differs from the test result of the actual test performed, the machine learning model is relearned. In another example, the machine learning model is relearned based on the fact that the accuracy rate of the test result output from the machine learning model in the judgment of the judgment unit 32 regarding the test result of the actual test performed has fallen below a standard level. In this case, for example, it is determined that the accuracy rate has fallen below the standard level based on the aforementioned accuracy rate falling below 90%. In yet another example, the machine learning model is relearned periodically at predetermined intervals, regardless of the aforementioned accuracy rate, etc.

[0056] Updating a machine learning model is performed in the same manner as generating a machine learning model. That is, updating a machine learning model also uses a large number of datasets, and in each dataset used, image data based on images before reflow and inspection results from the actual inspection performed after reflow are associated for the inspected printed circuit board. The learning model update unit 36 ​​then classifies the datasets into training datasets and evaluation datasets, and retrains the machine learning model using the training dataset in the same manner as model training in machine learning model generation. After retraining is complete, the retrained machine learning model is evaluated in the same manner as the evaluation of the trained model in machine learning model generation.

[0057] As described above, when the machine learning model is updated, the accuracy of the test results output by the machine learning model relative to the test results in the actual tests performed will be above the standard level. In updating the machine learning model, data is shown in one of the many datasets used for retraining in which the test results output by the machine learning model in the judgment of the judgment unit 32 differ from the test results in the actual tests performed after reflow. Therefore, the learning model update unit 36 ​​retrains the machine learning model in cases where the test results output by the machine learning model in the judgment of the judgment unit 32 differ from the test results in the actual tests performed after reflow.

[0058] In one example, when the inspection result output by the machine learning model in the judgment unit 32 is good, but the actual inspection result is bad, the machine learning model is retrained. In this case, in one of the many datasets, the image data input to the machine learning model for the good inspection result output and the inspection result from the actual inspection performed after reflow are shown. The image data input to the machine learning model for the good inspection result output is image data based on images I1, I2 and composite image I3, as described above.

[0059] Therefore, in the retraining of the machine learning model in this example, when a defect is newly discovered during inspection, the image data and inspection results based on images I1 to I3 are shown in one of the datasets. Then, during retraining, the machine learning model learns the features of the image data based on images I1 to I3 from the dataset showing the image data and inspection results when a defect is newly discovered during inspection. Note that the learning model update unit 36 ​​does not need to be provided in the information processing device 6. In one example, the machine learning model update process described above is performed by a computer or the like, separate from the information processing device 6.

[0060] As described above, in this embodiment, a composite image I3 is generated by combining the component with the pre-mounting image (e.g., I2) before the component is mounted on the substrate, suction information indicating the suction state of the component in the component mounting device 3, and setting information indicating the set position where the component is to be mounted on the substrate. Then, by inputting the pre-mounting image (e.g., I2) and the composite image I3 into a machine learning model that outputs inspection results in the post-reflow inspection from the input of image data based on the image before reflow, the appropriateness of mounting the component on the substrate at the set position set in the setting information is determined. For this reason, whether the component will be properly bonded to the substrate by mounting it at the set position is determined before reflow, taking into account not only the suction state of the component in the component mounting device 3 but also the state of the substrate before the component is mounted, such as the solder mounting state on the substrate. Thus, the appropriateness of the position for mounting the component on the substrate can be accurately determined before reflow.

[0061] Furthermore, in this embodiment, if it is determined that mounting a component at a set position is inappropriate based on the inspection results output from the machine learning model, the set position is changed in the setting information. Then, the composite image I3 is updated based on the changed set position, and the image data input to the machine learning model is updated based on the updated composite image I3. Then, based on the inspection results output from the machine learning model in response to the updated image data input, the appropriateness of mounting the component on the substrate at the changed set position is determined. Because the above processing is performed, a position in which the inspection results output from the machine learning model are good, that is, a position in which the component is properly bonded to the substrate, is appropriately searched for regarding the position in which the component is mounted on the substrate.Therefore, an appropriate position corresponding to the suction state in the component mounting device 3 and the state of the substrate before mounting the component is searched for regarding the position in which the component is mounted on the substrate.

[0062] Furthermore, the information processing device 6 places components on the substrate at the position searched by the machine learning model as a position where the inspection result output is good, using the component mounting device 3. This effectively suppresses the occurrence of defective printed circuit boards during post-reflow inspection. By suppressing the occurrence of defective printed circuit boards after reflow, material loss is appropriately reduced, and the increase in the number of processes due to repairing defective products in printed circuit board manufacturing is effectively suppressed. In addition, by suppressing the occurrence of defective printed circuit boards after reflow, the yield in printed circuit board manufacturing is improved.

[0063] Furthermore, in this embodiment, when generating a machine learning model, the model is trained using deep learning with training data that associates image data based on images before reflow with inspection results from actual inspections performed after reflow. As a result, a machine learning model is generated that appropriately outputs inspection results from post-reflow inspections, based on input image data based on pre-mounting images such as images I1 and I2, and composite image I3, which is created by combining the pre-mounting images with the components. Therefore, by using the machine learning model generated as described above, the appropriateness of mounting components on the substrate at the set position is determined, thereby more accurately determining the appropriateness of the position in which components are mounted on the substrate.

[0064] Furthermore, in this embodiment, when updating the machine learning model, if the inspection result output from the machine learning model differs from the inspection result in the actual inspection performed after reflow, the machine learning model is retrained. This retraining is performed using the image data based on the pre-mounting image (e.g., I2) and composite image I3 input to the machine learning model, as well as the inspection result in the actual inspection, when the inspection result output from the machine learning model differs from the inspection result in the actual inspection performed after reflow. As the machine learning model is updated as described above, the decrease in the accuracy of the inspection result output from the machine learning model relative to the inspection result in the actual inspection is effectively suppressed. Therefore, by using the machine learning model generated as described above to determine the appropriateness of mounting the component on the substrate at the set position, the appropriateness of the position in which the component is mounted on the substrate can be determined even more accurately.

[0065] (modified version) Figure 8 shows an example of the processing performed by the image processing unit 31, the determination unit 32, and the position search unit 33 in a modified example. The processing shown in Figure 8 is performed each time a substrate is brought into the manufacturing line. As shown in Figure 8, in this modified example as well, the processing from S101 to S105 is performed by the processing execution unit 21 in the same manner as in the embodiments described above. However, in this modified example, the number of changes N for which the set position has been changed is defined in the setting information. The determination unit 32 increments the number of changes N by 1 each time the set position indicated in the setting information is changed by the processing in S101. The number of changes N is reset to zero each time a substrate is brought into the manufacturing line.

[0066] In this modified example, if the inspection result output from the machine learning model in S104 is poor, that is, if it is not appropriate to mount the substrate on the component at the set position, the determination unit 32 makes a determination based on the number of changes N (S106). That is, the determination unit 32 determines whether the number of changes N in which the set position has been changed is equal to or greater than the reference number Nref. If the number of changes N is less than the reference number Nref, the position search unit 33 changes the set position set in the setting information (S101). On the other hand, if the number of changes N is equal to or greater than the reference number Nref, the determination unit 32 determines that the component will not be properly bonded to the substrate by reflow regardless of the position in which the component is mounted on the substrate (S107). That is, it is determined that the component will not be properly bonded to the substrate regardless of the mounting position of the component on the substrate.

[0067] If the determination in S107 is made, the processing execution unit 21, including the determination unit 32, generates a signal based on the determination result that the component is not properly bonded to the substrate, and outputs the generated signal. In one example, a signal warning that the component is not properly bonded to the substrate, regardless of the mounting position of the component on the substrate, is output from the processing execution unit 21, and a warning is issued using the user interface 25, etc. In another example, instead of a warning, or in addition to a warning, a command (command signal) to stop mounting the component to the substrate is output from the processing execution unit 21 to the component mounting device 3. Then, the component mounting device 3 stops the operation of mounting the component to the substrate based on the command from the information processing device 6.

[0068] This modified version also produces the same functions and effects as the embodiments described above. Furthermore, in this modified version, if the number of changes N in the setting information where the setting position has been changed is equal to or greater than the reference number Nref, it is determined that the component will not be properly bonded to the substrate by reflow, regardless of the position in which the component is mounted on the substrate. This effectively prevents the continuous search for an appropriate position for mounting the component on the substrate, in cases where the component will not be properly bonded to the substrate regardless of the mounting position of the component on the substrate.

[0069] Furthermore, in one modified version, even if the inspection result output from the machine learning model in processing S104, etc., is good, as long as the number of changes N in which the setting position has been changed is small compared to the reference number Nref, the position search unit 33 changes the setting position set in the setting information in processing S101, etc. Then, if there are multiple setting positions for which the inspection result is good, the position search unit 33, etc., selects the setting position with the largest margin to the setting position for which the inspection result is poor among the multiple setting positions for which the inspection result is good as the position for mounting the component. This modified version also produces the same functions and effects as the embodiments described above.

[0070] Furthermore, in the embodiments described above, image data input to the machine learning model is generated using three types of images I1 to I3: image I1 of the substrate only, image I2 of the substrate with only solder mounted, and composite image I3 of image I2 with components added. However, the invention is not limited to this. In one modified example, image data input to the machine learning model is generated using only two types of images: image I2 and composite image I3. In this case as well, composite image I3 is generated in the same manner as in the embodiments described above, based on image I2 (the image before mounting), adsorption information, and setting information. By inputting the image data based on image I2 and composite image I3 into the machine learning model, the appropriateness of mounting the components on the substrate at the setting positions set in the setting information is determined in the same manner as in the embodiments described above.

[0071] In one modified example, in the determination unit 32, in addition to image data based on the pre-mounting image (e.g., I2) and the composite image I3, information indicating the conditions in the reflow process is input to the machine learning model. In this case as well, the inspection results from the inspection to be performed after the reflow process are output from the machine learning model, and the determination unit 32, in the same manner as in the embodiments described above, determines the appropriateness of mounting the component on the substrate at the set position based on the inspection results output from the machine learning model. The information indicating the conditions in the reflow process includes the specifications of the reflow apparatus 4 and the ambient temperature of the environment in which the reflow process is performed.

[0072] In one modified example, in the determination by the determination unit 32, solder thickness information may be input to the machine learning model in addition to the image data based on the pre-mounting image (e.g., I2) and the composite image I3. In addition to the aforementioned image data based on the pre-mounting images (e.g., I1, I2) and the composite image I3, position information in the height direction (substrate and solder thickness direction) for each of the pre-mounting images may be input to the machine learning model. Position information in the height direction for each of the pre-mounting images may be, for example, an image of the height distribution for each of the pre-mounting images. In this modified example as well, the inspection results from the inspection to be performed after reflow are output from the machine learning model, and the determination unit 32, in the same manner as in the embodiments described above, determines the appropriateness of mounting the component on the substrate at the set position based on the inspection results output from the machine learning model.

[0073] According to at least one of these embodiments or examples, a composite image is generated by compositing a component onto a pre-mounting image based on a pre-mounting image of the substrate before the component is mounted, adsorption information indicating the adsorption state of the component in the component mounting device, and setting information indicating the set position where the component is mounted on the substrate. Then, the image data based on the pre-mounting image and the composite image is input to a machine learning model that outputs inspection results from post-reflow inspection based on image data based on the image before reflow, and the appropriateness of mounting the component at the set position is determined. This makes it possible to provide an information processing device and information processing method that can accurately determine the appropriateness of the position where a component is mounted on the substrate before reflow in the manufacturing of printed circuit boards by soldering.

[0074] While several embodiments of the present invention have been described, these embodiments are presented as examples only and are not intended to limit the scope of the invention. These novel embodiments can be carried out in a variety of other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims of the invention and its equivalents. [Explanation of symbols]

[0075] 1...Manufacturing system, 2...Solder printing machine, 3...Component mounting machine, 4...Reflow machine, 5...Inspection machine, 6...Information processing machine, 11-13...Photography machine, 15...Nozzle, 21...Processing execution unit, 22...Storage unit, 31...Image processing unit, 32...Determination unit, 33...Position search unit, 35...Learning model generation unit, 36...Learning model update unit, 51...Input layer, 52...Output layer, 53...Intermediate layer (hidden layer).

Claims

1. Information processing device for soldering components to a circuit board, An image processing unit generates a composite image by compositing the component onto the pre-mounting image, based on a pre-mounting image of the component before it is mounted on the substrate, adsorption information indicating the adsorption state of the component in the component mounting device, and setting information indicating the set position where the component is mounted on the substrate. A determination unit determines the appropriateness of mounting the component on the substrate at the set position set in the setting information by inputting the image data based on the image before reflow and the image data based on the composite image into a machine learning model that outputs the inspection result in the inspection after reflow from the input of image data based on the image before reflow, An information processing device equipped with the following:

2. The information processing apparatus according to claim 1, further comprising a position search unit that searches for a position on the substrate where the inspection result output from the machine learning model is good, based on the determination result from the determination unit.

3. When the inspection result output from the machine learning model is found to be defective, the position search unit changes the position set in the setting information as the position for mounting the component. The image processing unit updates the composite image based on the pre-mounting image and the adsorption information, as well as the setting information with the changed setting position. The determination unit inputs the image data based on the pre-mounting image and the updated composite image into the machine learning model to determine the appropriateness of mounting the component on the substrate at the setting position changed in the setting information. The information processing apparatus according to claim 2.

4. The determination unit determines that if the number of changes made to the setting position in the setting information is equal to or greater than a certain number of changes, the component will not be properly bonded to the substrate by reflow, regardless of the position in which the component is mounted on the substrate. The information processing apparatus according to claim 3.

5. The information processing apparatus according to claim 2, wherein the position search unit searches for a position where the inspection result output from the machine learning model is good, and then mounts the component on the substrate in the component mounting device at that position.

6. An information processing apparatus according to any one of claims 1 to 5, further comprising a learning model generation unit that trains a model by deep learning and generates a machine learning model using learning data which associates image data based on an image before reflow with inspection results from an inspection actually performed after the reflow.

7. The information processing apparatus according to any one of claims 1 to 5, further comprising a learning model update unit that updates the machine learning model by retraining the machine learning model using the image data based on the pre-installation image and the composite image input to the machine learning model, and the inspection results from the actual inspection performed, in the case where the inspection result output from the machine learning model differs from the inspection result in the inspection actually performed after the reflow.

8. The image processing unit uses an image of the substrate with only solder mounted on it as the pre-mounting image, and generates the composite image by compositing the components onto the image of the substrate with only solder mounted on it. The image processing unit generates the image data to be input to the machine learning model using the image of the substrate with only the solder mounted on it and the composite image. An information processing apparatus according to any one of claims 1 to 5.

9. The information processing apparatus of claim 8, wherein the image processing unit generates the image data to be input to the machine learning model using the image of the substrate with only the solder mounted on it and the composite image, as well as an image of the substrate alone.

10. A method for processing information related to soldering components to a circuit board, Based on a pre-mounting image of the substrate before the component is mounted, adsorption information indicating the adsorption state of the component in the component mounting device, and setting information indicating the set position where the component is mounted on the substrate, a composite image is generated by compositing the component onto the pre-mounting image. By inputting the image data based on the pre-mounting image and the image data based on the composite image into a machine learning model that outputs inspection results from the pre-reflow inspection based on the image before reflow, the appropriateness of mounting the component on the substrate at the set position set in the setting information is determined. An information processing method comprising the following:

Images