Information processing device and information processing method

Abstract

To provide an information processing device that enables determination of validity of a design in regard to soldering with high accuracy in a design stage before manufacturing is started.SOLUTION: According to an embodiment, an information processing device comprises: an image generation part; and a determination part. The image generation part generates input image data indication a state of before reflow on the basis of design information in regard to soldering, and applies a color or a pattern corresponded to color information set to each one type of a component in the input image data to each of a plurality of types of the components. The determination part inputs the input image data into a machine learning model by using the machine learning model that outputs an inspection result in inspection after the reflow from the input of the image data based on an image before the reflow, validity of the design indicated by the design information is determined.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] An embodiment of the present invention relates to an information processing device and an information processing method. [Background technology]

[0002] When joining components to a substrate such as a board by soldering, room temperature solder and the components are mounted on the substrate in that order. Then, with the solder and components mounted on the substrate, reflow is performed to melt the solder and join the components to the substrate by soldering. When a printed circuit board or the like is manufactured as a product in which components are joined to a substrate such as a board by soldering, the manufactured product is inspected after reflow, including visual inspection and electrical testing, to prevent defective products from being released onto the market.

[0003] As described above, when manufacturing a product by soldering, a design for soldering is performed before mounting solder and components on a base. In the design for soldering, a design for a base such as a circuit board, a design for a metal mask used for mounting solder on the base, and a design for components are performed. In manufacturing a product by soldering, in the design stage before the start of manufacturing, in order to prevent reworking of the design due to defects occurring during manufacturing, it is required that the appropriateness of the design for soldering be judged with high accuracy. For example, with respect to a design performed in real time, it is required that the inspection results after reflow when a product is manufactured based on the design be judged with high accuracy. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2020-161749 A Summary of the Invention [Problem to be solved by the invention]

[0005] The problem to be solved by the present invention is to provide an information processing apparatus and an information processing method that enable the appropriateness of a soldering design to be determined with high accuracy at the design stage before production begins. [Means for solving the problem]

[0006] According to an embodiment, an information processing device related to soldering of components to a base is provided. The information processing device includes an image generating unit and a determining unit, and the image generating unit generates input image data showing a state before reflow based on design information related to soldering. The image generating unit applies a color or pattern corresponding to the color information to each of the multiple types of components based on color information set for each type of component for the multiple types of components shown in the input image data. The determining unit uses a machine learning model that receives image data based on an image before reflow and outputs an inspection result in an inspection after reflow, and inputs the input image data in which the color or pattern corresponding to the color information is applied to each of the multiple types of components into the machine learning model to determine the appropriateness of the design shown in the design information. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 is a schematic diagram showing an example of a manufacturing system for manufacturing a product by soldering in the first embodiment. [Diagram 2] FIG. 2 is a block diagram illustrating an example of a configuration of an information processing device used in the design stage in the first embodiment. [Diagram 3] FIG. 3 is a schematic diagram illustrating an example of a machine learning model used for processing in the processing execution unit in the first embodiment. [Figure 4] FIG. 4 is a flowchart illustrating an example of a machine learning model generation process (construction process) performed by the learning model generation unit of the process execution unit in the first embodiment. [Diagram 5]FIG. 5 is a block diagram illustrating an example of processing performed by the image generating unit, the determining unit, and the color information setting unit in the information processing device according to the first embodiment. [Figure 6] FIG. 6 is a schematic diagram showing an example of a process for generating input image data to be input to a machine learning model from design information related to soldering in the first embodiment. [Figure 7] FIG. 7 is a flowchart illustrating an example of a process for identifying the cause of a defect in an inspection result, which is performed by a cause identifying unit of the process execution unit in the first embodiment. [Figure 8] FIG. 8 is a schematic diagram showing the types of data input to the machine learning model and the accuracy rates of the output results from the machine learning model for the test results in actual tests in each of Example 1, Comparative Example 1, Comparative Example 2, and Comparative Example 3 in verification related to the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] Hereinafter, embodiments will be described with reference to the drawings.

[0009] (First embodiment) First, a first embodiment will be described as an example of an embodiment. Fig. 1 shows an example of a manufacturing system 1 that manufactures products by soldering in the first embodiment. As shown in Fig. 1 etc., the manufacturing system 1 includes a solder mounting device 2, a component mounting device 3, a reflow device 4, and an inspection device 5. In the manufacturing system 1, a manufacturing line for manufacturing products is formed, and in the manufacturing line, the solder mounting device 2, the component mounting device 3, the reflow device 4, and the inspection device 5 are arranged in this order from the upstream side.

[0010] In the manufacturing line, a base is carried into the solder mounting device 2. Examples of the base carried into the solder mounting device 2 include a substrate and a lead frame. The substrate carried into the solder mounting device 2 as the base may be a printed wiring board, or a printed circuit board (printed circuit board) on which components such as chip components have already been attached to a printed wiring board. The solder mounting device 2 mounts solder on the substrate to be the base, for example, by printing solder on pads or lands formed on the surface of the substrate. In one example, solder may be printed on the surface of a component already attached to the printed circuit board to mount solder on the substrate to be the base. In the solder mounting device 2, solder at room temperature is mounted on the substrate, and the solder is mounted on the substrate in an unmolten state.

[0011] Here, in the solder mounting device 2, with the surface of the base covered with a metal mask, the openings in the metal mask are filled with solder paste, and the solder is transferred to the pads of the base through the openings in the metal mask. Then, by removing the metal mask from the base, only the solder transferred through the openings in the metal mask is left mounted on the base. In this way, the solder is printed on the base. Note that the method of mounting the solder on the base in the solder mounting device 2 is not limited to printing. In one example, the solder may be mounted on the base by either dispensing (applying) the solder or mounting a solder sheet.

[0012] A base body on which solder has been mounted is carried into the component mounting device 3. The component mounting device 3 mounts components to be newly attached to a base body such as a substrate. Examples of components to be mounted on the base body include chip components, IC packages, and connector components. By mounting the components, the solder is sandwiched between the base body and the components.

[0013] A base on which solder and components are mounted is carried into the reflow device 4. The reflow device 4 performs soldering by reflow. The solder mounted on the base, such as a board, is melted by reflow, and the newly mounted components are joined to the base. In this way, a printed circuit board or the like is formed as a product in which the mounted components are attached to the base. In one example, the newly mounted components are joined by soldering to pads or lands formed on the surface of the base, which is the board. In another example, the newly mounted components are joined by soldering to components already mounted on the base, such as a board.

[0014] Manufactured products such as printed circuit boards, i.e., products in which components are bonded to a base by reflow in the reflow device 4, are carried into the inspection device 5. The inspection device 5 inspects the manufactured products such as printed circuit boards and judges whether the products are good or bad. Only products that are judged to be good by the inspection in the inspection device 5 are released onto the market as distribution goods.

[0015] For example, the inspection device 5 performs an appearance inspection and an electrical test on a manufactured product such as a printed circuit board. In the appearance inspection, an image of a product in which components are joined to a base by reflow is obtained by photography or the like, and the manufactured product such as a printed circuit board is judged as good or bad based on the obtained product image. In the electrical test, electricity is passed through the manufactured product and the amount of electricity is measured by a tester, and the product such as a printed circuit board is judged as good or bad.

[0016] Moreover, in the example manufacturing system 1 of FIG. 1, photographing devices 11 to 13 are provided. Each of the photographing devices 11 to 13 is, for example, a camera or a video camera. The photographing device 11 is disposed upstream of the solder mounting device 2, and photographs an image of only the substrate. The photographing device 12 is disposed between the solder mounting device 2 and the component mounting device 3, and photographs an image of only the solder mounted on the substrate. The photographing device 13 is disposed between the component mounting device 3 and the reflow device 4, and photographs an image of the substrate with solder and components mounted. Thus, in the example of FIG. 1, the photographing devices 11 to 13 photograph three types of images, an image of only the substrate, an image of only the substrate with solder mounted on the substrate, and an image of the substrate with solder and components mounted on the substrate, as images before reflow by the reflow device 4.

[0017] When manufacturing a product such as a printed circuit board by soldering as described above, a design for soldering is performed before mounting solder and components on a base. In the design for soldering, a base such as a board is designed, a metal mask used for mounting solder on the base is designed, and components are designed. In this embodiment, in the design stage before the solder is mounted on the base, i.e., before manufacturing begins, the following processing is performed using an information processing device 6.

[0018] Fig. 2 shows an example of the configuration of the information processing device 6 used in the design stage in this embodiment. As shown in Fig. 2, the information processing device 6 includes a processing execution unit 21, a storage unit 22, a communication interface 23, and a user interface 25. In the example shown in Fig. 2, the processing execution unit 21 includes an image generation unit 31, a judgment unit 32, a color information setting unit 33, a learning model generation unit 35, and a cause identification unit 36. Each of the image generation unit 31, the judgment unit 32, the color information setting unit 33, the learning model generation unit 35, and the cause identification unit 36 ​​performs a part of the processing performed by the processing execution unit 21.

[0019] In one example, the information processing device 6 is configured with a computer or the like, and a processor or integrated circuit of the computer functions as the processing execution unit 21. The processor or integrated circuit of the computer includes any one of a central processing unit (CPU), an application specific integrated circuit (ASIC), a graphics processing unit (GPU), a microcomputer, a field programmable gate array (FPGA), and a digital signal processor (DSP). The computer serving as the information processing device 6 may include only one integrated circuit or the like, or may include multiple integrated circuits or the like.

[0020] In addition, in a computer serving as the information processing device 6, a storage medium of the computer functions as the storage unit 22. The storage medium may include a main storage device such as a memory, as well as an auxiliary storage device. Examples of the storage medium include a magnetic disk, an optical disk (CD-ROM, CD-R, DVD, etc.), a magneto-optical disk (MO, etc.), and a semiconductor memory. The computer serving as the information processing device 6 may include only one storage medium or multiple storage media.

[0021] In the computer serving as the information processing device 6, a processor or integrated circuit executes a program stored in a storage medium or the like, thereby performing the processing described below by a processing execution unit 21. For example, the processor or the like executes an information processing program stored in a storage medium or the like, thereby performing the processing described below. In one example, in the computer serving as the information processing device 6, the program executed by the processor or the like may be stored in a computer (server) connected via a network such as the Internet, or in a server in a cloud environment. In this case, the processor downloads the program via the network.

[0022] In addition, in one example, the information processing device 6 is configured with a plurality of computers that are separate from each other. In this case, the process execution unit 21 performs the process described below by processors or integrated circuits of the plurality of computers.

[0023] In one example, the information processing device 6 is configured as a server in a cloud environment. The infrastructure of the cloud environment is configured by a virtual processor such as a virtual CPU and a cloud memory. In the server in the cloud environment that is the information processing device 6, the virtual processor or the like functions as a process execution unit 21, and the process described below is performed by the process execution unit 21 by the virtual processor or the like. And the cloud memory functions as a storage unit 22.

[0024] In the information processing device 6, the communication interface 23 is configured as an interface for accessing an external device. The information processing device 6 is capable of communicating with the external device via the communication interface 23 in a wired or wireless manner.

[0025] In addition, in the information processing device 6, various operations are inputted in a user interface 25 by users of the information processing device 6, including a designer who designs soldering. The user interface 25 is provided with any of buttons, switches, touch panels, etc. as operation members through which operations are inputted by users of the information processing device 6. In addition, the user interface 25 notifies users of the information processing device 6 of various information. The information is notified, for example, by displaying a screen or emitting a voice. In one example, the user interface 25 is provided as an external device of the information processing device 6, and is provided separately from a computer constituting the information processing device 6.

[0026] The following describes the processing performed by the processing execution unit 21. As shown in Fig. 2 etc., a machine learning model 40 generated (constructed) as described below is stored in the storage unit 22. The processing execution unit 21 uses the machine learning model 40 to perform processing.

[0027] 3 shows an example of a machine learning model 40 used for processing in the processing execution unit 21. As shown in FIG. 3 and other figures, the machine learning model 40 is composed of an input layer 41, an output layer 42, and an intermediate layer (hidden layer) 43 between the input layer 41 and the output layer 42. In the machine learning model 40, in response to image data based on an image before reflow by the reflow device 4 being input to the input layer 41, an inspection result of the inspection by the inspection device 5 after reflow is output from the output layer 42.

[0028] The image data input to the input layer 41 of the machine learning model 40 is generated based on a plurality of types of images showing the state before reflow, for example, by arranging a plurality of types of images showing the state before reflow. In one example, the image data input to the machine learning model 40 is generated by arranging three types of images: an image of only a base such as a board, an image of only solder mounted on the base, and an image of solder and components mounted on the base.

[0029] When the above-mentioned image data based on the image before reflow is input to the machine learning model 40, pixel information such as RGB values ​​for each pixel for all images constituting the image data is input to the input layer 41. For example, when image data in which multiple types of images showing the state before reflow are arranged as described above is input to the machine learning model 40, pixel information for each pixel for all of the multiple types of images arranged is input to the input layer 41.

[0030] 3, an input layer 41 is configured from nodes the same number as the total number of pixels in all images constituting the input image data. In the example of FIG. 3, the total number of pixels in all images constituting the input image data is k, and pixel information x1 to xk is input to the input layer 41 by inputting the image data to the machine learning model 40.

[0031] 3, whether the inspection after reflow is pass or fail is output from the output layer 42. For this reason, in the machine learning model 40, the output layer 42 is configured from a node that outputs that the inspection is pass and a node that outputs that the inspection is fail.

[0032] The intermediate layer 43 is composed of a convolution layer, a pooling layer, a fully connected layer, and the like. In the convolution layer, a filter process is performed for each pixel to extract features in the image. In the pooling layer, the image is reduced while maintaining the features of the image. The processing in the pooling layer absorbs positional deviations of the features in the image constituting the image data input to the machine learning model 40. In the intermediate layer 43, the features of the image are recognized by processing in multiple convolution layers and multiple pooling layers.

[0033] In addition, the fully connected layer converts the image data, whose features have been recognized by the convolution layer and the pooling layer, into one-dimensional data. By performing the above-described processing, the intermediate layer 43 of the machine learning model 40 recognizes features in the image. The recognized features are then converted into information used to determine pass / fail in the inspection.

[0034] In this embodiment, the machine learning model 40 is generated (constructed) by the learning model generating unit 35. In generating the machine learning model 40, learning data consisting of a large number of data sets is used. In each data set of the learning data, image data based on a plurality of types of images before reflow is shown for a product such as a printed circuit board that has been inspected in the past. The image data shown in each data set is generated, for example, by arranging a plurality of types of images showing a state before reflow. In one example, the image data of each data set of the learning data is generated based on photography of a state before reflow. In this case, for example, the image data of each data set is generated by arranging three types of photographed images photographed by the photographing devices 11 to 13.

[0035] Furthermore, each of the data sets of the learning data shows the inspection results of the inspection actually performed after reflow for products such as printed circuit boards that were previously inspected. Each of the data sets may show the image data and inspection results described above for products previously inspected in the manufacturing system 1, or may show the image data and inspection results described above for products previously inspected in another manufacturing system having a similar configuration to the manufacturing system 1.

[0036] Since the learning data is constructed as described above, in each of the many data sets of the learning data, image data based on images before reflow is associated with the inspection results of the inspection actually performed after reflow for the products inspected in the past. Note that the inspected products differ from one another among the many data sets.

[0037] Fig. 4 shows an example of a generation process (construction process) of a machine learning model 40 by the learning model generation unit 35. When the process of Fig. 4 starts, the learning model generation unit 35 classifies a large number of data sets of learning data into a training data set and an evaluation data set (S101). In one example, a large number of data sets are classified in a state where the ratio of the training data set to the evaluation data set is 9.

[0038] Then, the learning model generation unit 35 uses the training data set of the learning data to train a model by deep learning (S102). At this time, for example, a neural network is trained as the model. The model is also trained by supervised learning in which the inspection results shown in each of the training data sets are given as correct answers. By deep learning, the model learns the features included in the image data of the training data set, for example, learning the features in the image data when the inspection result is bad.

[0039] When the learning using the training data set is completed, the learning model generation unit 35 evaluates the learned model using the evaluation data set (S103). In the model evaluation, for each evaluation data set, image data based on the image before reflow is input to the learned model. Then, for each evaluation data set, the output result from the learned model is compared with the inspection result of the actual inspection. Then, as an index for evaluating the learned model, the accuracy rate of the output result from the model with respect to the inspection result of the actual inspection is calculated.

[0040] When the evaluation using the evaluation data set is completed, the learning model generation unit 35 determines whether the accuracy rate calculated as an index for evaluating the learned model is equal to or higher than a reference level (S104). In one example, the accuracy rate is determined to be equal to or higher than the reference level based on the accuracy rate being 90% or higher. If the accuracy rate is equal to or higher than the reference level (S104-Yes), the learning model generation unit 35 stores the learned model in the storage unit 22 as the above-mentioned machine learning model 40 to be used for processing (S105).

[0041] If the accuracy rate is lower than the reference level (S104-No), the learning model generation unit 35 adds a data set to the learning data (S106). Then, the process returns to S121, and the learning model generation unit 35 sequentially performs the processes from S101 onwards. This allows learning of a model using the added data set and evaluation of the model using the added data set. Note that the learning model generation unit 35 does not necessarily need to be provided in the information processing device 6. In one example, the generation process (construction process) of the machine learning model 40 described above is performed by a computer or the like other than the information processing device 6.

[0042] 5 shows an example of processing in the image generating unit 31, the determining unit 32, and the color information setting unit 33 in the information processing device 6 according to this embodiment. The processing shown in FIG. 5 is performed, for example, every time design information related to soldering is generated in the design stage before the start of manufacturing. As shown in FIG. 4, the image generating unit 31 generates input image data showing a state before reflow as data to be input to the above-mentioned machine learning model 40 (S110). Design information related to soldering is input to the image generating unit 31, and the image generating unit 31 generates the input image data based on the input design information.

[0043] To the image generating unit 31, for example, CAD (Computer Aided Design) data is input as design information related to soldering. Moreover, as the design information related to soldering, design information on a base such as a substrate, design information on a metal mask used to mount solder on the base, and design information on components to be mounted are shown. Moreover, in the input image data, a plurality of types of images before reflow are shown, and for example, a plurality of types of images showing a state before reflow are arranged. In one example, in the input image data, three types of images are arranged as images showing a state before reflow, namely, an image of only a base such as a substrate, an image with only solder mounted on the base, and an image with solder and components mounted on the base.

[0044] In addition, the input image data shows a plurality of types of components in a plurality of types of images of a state before reflow. The plurality of types of components shown in the input image data include solder and parts, as well as one or more types of components constituting a base. Components constituting a base such as a substrate include resist, pads, lands, and isolation parts.

[0045] The color information setting unit 33 sets color information for each of the multiple types of components represented in the input image data (S111). Color information is set for each of the multiple types of components. Different color information is set for each of the multiple types of components represented in the input image data. The multiple types of components are distinguished from each other by the color information, and each of the multiple types of components is specified by the color information. Therefore, each of the multiple types of components is associated with corresponding color information.

[0046] For example, captured images of multiple types of components indicated by input image data are input to the color information setting unit 33. In one example, captured images of a product previously inspected before reflow are input to the color information setting unit 33, and for example, captured images previously captured by the photographing devices 11 to 13 are input to the color information setting unit 33. The color information setting unit 33 specifies a color or pattern for each of the multiple types of components based on the input captured images. Then, the color information setting unit 33 sets color information indicating the specified color or pattern for each of the multiple types of components. The color information for each of the multiple types of components is input to the image generating unit 31.

[0047] In generating input image data (S110), the image generating unit 31 generates line diagram data based on design information related to soldering (S112). In the line diagram data, multiple types of line diagrams (line diagram images) showing the state before reflow are arranged. In one example, in the line diagram data, three types of line diagrams are arranged as line diagrams (line diagram images) showing the state before reflow: a line diagram of only a base such as a board, a line diagram of a base with only solder mounted thereon, and a line diagram of a base with solder and components mounted thereon. In the line diagram data, multiple types of components are also shown, and the same components as those shown in the input image data are shown. However, in the line diagram data, each of the multiple types of components is not given a color or a pattern.

[0048] In generating line diagram data (S112), image generating unit 31 generates, in the line diagram data, outline elements of each of the multiple types of components based at least on the design information related to soldering. The outline elements of each of the multiple types of components are shown by lines, and are not colored or patterned. In each of the multiple types of line diagrams shown in the line diagram data, the outline elements of each of the components are shown at positions corresponding to the design information related to soldering, and are shown in shapes that at least correspond to the design information related to soldering. As the outline elements of the multiple types of components, outline elements of one or more types of components constituting the base, outline elements of solder, and outline elements of parts to be mounted are generated.

[0049] The image generating unit 31 forms the respective contour elements of one or more types of components constituting the base based at least on the design information for the base. In the design information for the base, the design shape of each of the components constituting the base is indicated. Then, each contour element of the component constituting the base is formed into a shape corresponding to the design shape of that component in the design information for the base. For example, the contour element of a pad is formed into a shape corresponding to the design shape of the pad indicated in the design information for the base. In each of the multiple types of line drawings indicated by the line drawing data, each contour element of one or more types of components constituting the base is indicated at a position corresponding to the design information for the base.

[0050] Furthermore, the image generating unit 31 forms an external element of a component to be mounted on the base based on design information about the component. The design information about the component indicates a design shape of the component. The external element of the component, which is one type of component among a plurality of types, is formed into a shape corresponding to the design shape of the component in the design information about the component. In the line diagram data, the external element of the component is indicated at a position corresponding to the design information about the component in a line diagram in which the solder and the component are mounted on the base.

[0051] The image generating unit 31 also forms an outline element of the solder based on design information about a metal mask used to mount the solder on the substrate. The design information about the metal mask indicates the design shape of the opening of the metal mask. The outline element of the solder, which is one type of component among the multiple types of components, is formed in a shape that at least corresponds to the design shape of the opening in the design information about the metal mask. In the line diagram data, the outline element of the solder is indicated at a position corresponding to the design information about the metal mask in each of a line diagram in which only the solder is mounted on the substrate and a line diagram in which the solder and components are mounted on the substrate.

[0052] Here, when the solder is mounted on the substrate by transferring the solder through the openings of the metal mask, some of the solder filled in the openings may adhere to the metal mask when it is removed from the substrate, or some of the solder filled in the openings may drip toward the substrate. For this reason, the shape of the solder mounted on the substrate is not necessarily the same as the shape of the openings of the metal mask. In the example of FIG. 5, the image generating unit 31 generates the external shape elements of the solder based on the design information about the metal mask as well as the relationship data indicating the relationship between the openings of the metal mask and the shape of the solder after it is mounted on the substrate. The relationship data is stored in the storage unit 22, for example.

[0053] In the example of FIG. 5, the image generating unit 31 corrects the design shape of the opening indicated in the design information of the metal mask based on the relationship data indicating the relationship between the opening of the metal mask and the shape of the solder after mounting on the base. Then, the image generating unit 31 generates the outline element of the solder by correcting the design shape of the opening in correspondence with the relationship data. Therefore, in the example of FIG. 5, the outline element of the solder, which is one type of the multiple types of components, is formed into a shape corresponding to the design shape of the opening in the design information of the metal mask and the relationship data indicating the relationship between the opening of the metal mask and the shape of the solder after mounting on the base. Then, in the example of FIG. 5, the line diagram data is generated using the outline element of the solder generated based on the design information of the metal mask and the relationship data.

[0054] In one example, a solder outline element is formed without using the above-mentioned relationship data. In this case, for example, a solder outline element, which is one of the multiple types of components, is formed without correcting the design shape of the opening in the design information for the metal mask.

[0055] In generating input image data (S110), the image generating unit 31 generates line drawing data, and then performs coloring or the like on each of the multiple types of components indicated by the line drawing data in a color corresponding to the color information set by the color information setting unit 33 (S113). That is, the image generating unit 31 applies a color or pattern corresponding to the set color information to the external elements for each of the multiple types of components indicated by the line drawing data. In this way, input image data is generated from the line drawing data in which the multiple types of components are indicated by lines only, in which the multiple types of components are applied with a color or pattern corresponding to the color information.

[0056] In one example, only one predetermined color or pattern corresponding to the color information is set for the color or pattern to be applied to each of the multiple types of components. Then, in the input image data, the multiple types of components are each applied with the set predetermined color or pattern. In another example, a color range or pattern range corresponding to the color information is set for the color or pattern to be applied to each of the multiple types of components. Then, in the input image data, each of the multiple types of components is applied with a color included in the set color range or a pattern included in the set pattern range. However, in either case, in the input image data, the multiple types of components are identified based on the color or pattern, and the color or pattern is applied to each of the multiple types of components in a state in which each of the multiple types of components is specified based on the color or pattern.

[0057] In the input image data, a color or pattern corresponding to the color information of the solder is applied to the portion where the solder overlaps with the board, and a color or pattern corresponding to the color information of the component is applied to the portion where the component overlaps with at least one of the solder and the board.

[0058] The determination unit 32 uses the above-mentioned machine learning model 40 to determine the appropriateness of the design indicated by the design information related to soldering (S114). In determining the appropriateness of the design, the determination unit 32 inputs the input image data generated by the image generation unit 31 to the input layer 41 of the machine learning model 40. Then, the determination unit 32 determines the appropriateness of the design based on the output result output from the output layer 42 of the machine learning model 40 in response to the input of the input image data. That is, the determination unit 32 causes the machine learning model 40 to output the inspection result of the inspection after reflow in response to the input of the input image data, and determines the appropriateness of the design based on the inspection result output from the machine learning model 40.

[0059] When the inspection result output from the machine learning model 40 is good, the determination unit 32 determines that the design indicated by the design information is appropriate. On the other hand, when the inspection result output from the machine learning model 40 is bad, the determination unit 32 determines that the design indicated by the design information is inappropriate. The determination unit 32 outputs a determination result regarding the appropriateness of the design indicated by the design information regarding soldering. In one example, the process execution unit 21 causes the user interface 25 to notify the determination result by the determination unit 32.

[0060] Here, in the input image data input to the machine learning model 40, a color or pattern corresponding to the color information is applied to each of the multiple types of components. The multiple types of components are distinguished from one another by the color information, and each of the multiple types of components is specified by the color information. Therefore, the machine learning model 40 specifies each of the multiple types of components based on the color or pattern in each of the multiple types of images shown in the input image data.

[0061] For example, assume that the multiple types of components shown in the input image data include components C1 and C2, and color information α1 and α2 are set for the components C1 and C2, respectively. In this case, the machine learning model 40 identifies a portion of the input image data that has a color or pattern corresponding to the color information α1 as the component C1, and identifies a portion of the input image data that has a color or pattern corresponding to the color information α2 as the component C2.

[0062] FIG. 6 shows an example of a process of generating input image data to be input to a machine learning model from design information related to soldering in this embodiment. In the example of FIG. 6, CAD data Dc is used as design information related to soldering. In the CAD data Dc, a board shape 51 is shown as a portion corresponding to the entire board, which is a base. In the CAD data Dc, a design shape 52 of a pad and a design shape 53 of an isolation part are shown on the board shape 51 as design information for the board. In the CAD data Dc, a part of the board shape 51 other than the design shape 52 of the pad and the design shape 53 of the isolation part are designed as a resist.

[0063] In addition, in the CAD data Dc, a design shape 55 of an opening of a metal mask is shown as design information for the metal mask. In the CAD data Dc, a design shape 56 of a part is shown as design information for the part. In the CAD data Dc, the design shapes 52-53 are shown by lines only, and no color or pattern is applied to the design shapes 52-56.

[0064] 6, line diagram data Db is generated based at least on CAD data Dc. In the line diagram data Db, a line diagram Ib1 of only the substrate, which is the base, a line diagram Ib2 of the substrate with only solder mounted thereon, and a line diagram Ib3 of the substrate with solder and components mounted thereon are arranged.

[0065] In each of the line drawings (line drawing images) Ib1 to Ib3, a board shape 61 is shown as a portion corresponding to the entire board, which is the base. In each of the line drawings Ib1 to Ib3, a pad outline element 62, an isolation portion outline element 63, and a resist outline element 64 are shown as outline elements of the components that make up the board. In each of the line drawings Ib1 to Ib3, the portion of the board shape 61 other than the pad outline element 62 and the isolation portion outline element 63 becomes the resist outline element 64. Each of the outline elements 62 to 64 is generated based at least on design information about the board, and is formed into a shape that at least corresponds to the pad design shape 52, the isolation portion design shape 53, etc.

[0066] Moreover, in the line diagram data Db, a solder outline element 65 is shown in each of the line diagrams Ib2 and Ib3. The solder outline element 65 is generated based at least on design information about the metal mask, and is formed into a shape that at least corresponds to the design shape 55 of the opening of the metal mask. The solder outline element 65 may be generated based on relationship data that indicates the relationship between the opening of the metal mask and the shape of the solder after mounting on the board, in addition to the design information about the metal mask. Furthermore, in the line diagram data Db, a component outline element 66 is shown in the line diagram Ib3. The component outline element 66 is generated based at least on design information about the component, and is formed into a shape that at least corresponds to the design shape 56 of the component.

[0067] 6, input image data Da is generated by applying a color or pattern corresponding to the color information to each of multiple types of components in line diagram data Db. In the input image data Da, an image Ia1 of only a substrate, which is a base, an image Ia2 of a substrate with only solder mounted thereon, and an image Ia3 of a substrate with solder and components mounted thereon are arranged. Images Ia1, Ia2, and Ia3 are generated by applying a color or pattern corresponding to the color information to each of the components in line diagrams Ib1, Ib2, and Ib3, respectively.

[0068] In the example of FIG. 6, the pad outline element 62 is colored orange in correspondence with the color information set for the pad, and the isolation portion outline element 63 is colored black in correspondence with the color information set for the isolation portion. The resist outline element 64 is colored green in correspondence with the color information set for the resist, and the solder outline element 65 is colored gray in correspondence with the color information set for the solder. In the component outline element 66, the portion 67 is colored yellow, and the portion other than the portion 67 is colored white in correspondence with the color information set for the component. The portion where the solder overlaps with the board is colored in correspondence with the color information of the solder, and the portion where the component overlaps with at least one of the solder and the board is colored in correspondence with the color information of the component.

[0069] In addition, in this embodiment, when the machine learning model 40 outputs that the inspection result after reflow is defective due to the input of the input image data to the machine learning model 40, the cause identification unit 36 ​​identifies the cause of the defective inspection result. At this time, the cause identification unit 36 ​​identifies the cause of the defect using XAI (explainable artificial intelligence) technology. The cause of the defective inspection result is identified in one or more of the input image data input to the machine learning model 40, the design information such as CAD data used to generate the input image data, and the data generated in the process of generating the input image data from the design information. Note that the data generated in the process of generating the input image data from the design information includes the above-mentioned line diagram data.

[0070] 7 shows an example of a process for identifying the cause of a defect in an inspection result, which is performed by the cause identifying unit 36. The process of the example of FIG. 7 is performed when the machine learning model 40 outputs that the inspection result after reflow is defective, that is, when the determining unit 32 determines that the design indicated by the design information regarding soldering is not appropriate. When the process of the example of FIG. 7 is started, the cause identifying unit 36 ​​extracts nodes that have a high degree of contribution to a defective inspection result in the intermediate layer 43 and the input layer 41 of the machine learning model (S121). That is, nodes that have a high degree of contribution to the output result from the node that outputs that the output layer 42 is defective are extracted from the intermediate layer 43 and the input layer 41.

[0071] Then, the cause identification unit 36 ​​identifies a portion related to the node extracted as a node having a high degree of contribution to the defect determination in one or more of the input image data input to the machine learning model 40, the design information used to generate the data in the input image, and the data generated in the process of generating the input image data from the design information (S122). As a result, the portion related to the extracted node in one or more of the input image data, the design data, and the data generated in the process of generating the input image data is identified as the cause of the defect.

[0072] Then, the cause identifying unit 36 ​​notifies the part identified as the cause of the defect using the user interface 25 or the like (S123). At this time, in one example, the part identified as the cause of the defect is notified by displaying it in the input image data or the CAD data serving as the design information, for example by surrounding the part identified as the cause of the defect with a frame. For example, when the external element of a certain pad is surrounded by a frame in the input image data, or when the design shape of a certain pad is surrounded by a frame in the design information, the design of the pad surrounded by the frame can be recognized as the cause of the inspection result being defective. Note that, in one example, the above-mentioned process by the cause identifying unit 36 ​​does not need to be performed.

[0073] As described above, in this embodiment, input image data showing a state before reflow is generated based on design information related to soldering, and a color or pattern corresponding to the color information is applied to each of the multiple types of components based on color information set for each type of component for the multiple types of components shown in the input image data. Then, using a machine learning model 40 that inputs image data based on an image before reflow and outputs an inspection result in an inspection after reflow, the input image data in which a color or pattern corresponding to the color information is applied to each of the multiple types of components is input to the machine learning model 40, and the appropriateness of the design shown in the design information is determined. At the design stage before manufacturing begins, the appropriateness of the design related to soldering can be determined with high accuracy. For example, with respect to a design performed in real time, the inspection result after reflow when a product is manufactured based on the design can be determined with high accuracy.

[0074] In addition, in this embodiment, color information is set for each of the multiple types of components such that the multiple types of components are distinguished from each other by the color information and each of the multiple types of components is specified by the color information. Therefore, by applying a color or pattern corresponding to the color information to each of the multiple types of components shown in the input image data, the machine learning model 40 can appropriately specify each of the multiple types of components based on the color or pattern in the input image data. This further improves the accuracy of the judgment using the machine learning model 40 regarding the appropriateness of the soldering design.

[0075] In the present embodiment, in generating the input image data, external elements of each of the multiple types of components are generated based at least on design information related to soldering. Then, in generating the input image data, a color or pattern corresponding to the set color information is applied to the generated external elements for each of the multiple types of components. In this way, input image data in which a color or pattern corresponding to the color information is applied to each of the multiple types of components is appropriately generated using the design information.

[0076] In one example of this embodiment, in generating input image data, a contour element of the solder is formed based on relationship data that indicates a relationship between an opening of the metal mask and a shape of the solder after mounting on a substrate, in addition to design information about the metal mask. This allows a contour element to be more appropriately formed for the solder, which is one type of component among multiple types.

[0077] In the present embodiment, in generating the machine learning model 40, the model is trained by deep learning using learning data in which image data based on an image showing a state before reflow is associated with the inspection results of an inspection actually performed after reflow. Therefore, the machine learning model 40 is generated, which appropriately outputs the inspection results of the inspection after reflow from the input of image data based on an image before reflow.

[0078] Furthermore, in one example of this embodiment, when the machine learning model 40 outputs that the inspection result after reflow will be defective due to input of input image data to the machine learning model 40, the cause of the defective inspection result is identified in one or more of the input image data, the design information used to generate the input image data, and the data generated in the process of generating the input image data from the design information. This allows the designer or the like to quickly and appropriately find out which design element in the soldering design caused the defective inspection result.

[0079] (Modification) In the above-described embodiment, color information is set for each of the multiple types of components based on captured images of the multiple types of components. However, in a modified example, the color information setting unit 33 sets color information for each of the multiple types of components based on an operation input by a designer or the like on the user interface 25. In this case, for example, the designer or the like recognizes the color or pattern of each of the multiple types of components from the base, solder, parts, etc. that are actually used. Then, the designer or the like inputs the recognized color or pattern for each of the multiple types of components by an operation input on the user interface 25. Then, the color information setting unit 33 sets color information indicating the color or pattern input by the designer or the like for each of the multiple types of components.

[0080] However, in this modification, the multiple types of components are also distinguished from one another by color information, and each of the multiple types of components is identified by color information. Therefore, in this modification, the machine learning model 40 also identifies each of the multiple types of components based on the color or pattern in each of the multiple types of images represented by the input image data.

[0081] In the above embodiment, three types of images are generated as images showing the state before reflow in the input image data: an image of only the base, an image with only solder mounted on the base, and an image with solder and components mounted on the base, but the present invention is not limited to this. In a modified example, only two of the above three types of images before reflow are used to generate input image data to be input to the machine learning model 40. Therefore, the input image data to be input to the machine learning model 40 may show multiple types of images of the state before reflow.

[0082] In a modified example, in addition to the above-mentioned input image data showing the state before reflow, information showing the conditions for reflow is input to the machine learning model 40. In this case, too, the input image data and the like are input to the machine learning model 40, and the inspection results of the inspection after reflow are output from the machine learning model 40. Then, similarly to the above-mentioned embodiment and the like, the appropriateness of the design shown in the design information regarding soldering is determined based on the output results from the machine learning model 40. The information showing the conditions for reflow includes specification information of the reflow device 4 and the environmental temperature of the environment in which reflow is performed, and the like.

[0083] In a modified example, in addition to the above-mentioned input image data showing the state before reflow, at least one of solder thickness information and position information about the height direction (thickness direction of the base, solder, and component) in each of the images shown in the input image data is input to the machine learning model. As the position information about the height direction in each of the images shown in the input image data, for example, the height distribution in each of the images is shown. In this modified example, the input image data and the like are input to the machine learning model 40, and the inspection result in the inspection after reflow is output from the machine learning model 40. Then, similar to the above-mentioned embodiment, the appropriateness of the design shown in the design information regarding soldering is determined based on the output result from the machine learning model 40.

[0084] (Verification related to the embodiment) The following verification was performed as a verification related to the embodiment. In the verification, a machine learning model similar to the above-mentioned machine learning model 40 was used. That is, the verification was performed using a machine learning model that inputs image data based on an image before reflow and outputs an inspection result in an inspection after reflow.

[0085] In the verification, a plurality of printed circuit boards manufactured as products by soldering were used, and five printed circuit boards manufactured as products were actually used. Each of the plurality of printed circuit boards was inspected by an inspection device similar to the inspection device 5. Furthermore, for each of the plurality of printed circuit boards, a plurality of types of photographed images showing the state before reflow were taken. In fact, for each of the plurality of printed circuit boards, three types of photographed images were taken: an image of only the substrate, which is the base, an image of the substrate with only solder mounted thereon, and an image of the substrate with both solder and components mounted thereon. Note that, for each of the plurality of printed circuit boards, a substrate consisting of a resist, a pad, and an isolation portion was used as the substrate.

[0086] In the verification, data to be input to the machine learning model was generated for each of the multiple printed circuit boards using the three types of captured images shown before reflow. The generated data for each of the multiple printed circuit boards was then input into the machine learning model, and the inspection results after reflow were output from the machine learning model. The output results from the machine learning model were then compared with the inspection results from the actual inspection for each of the multiple printed circuit boards. The accuracy rate of the output results from the machine learning model relative to the inspection results from the actual inspection was then calculated.

[0087] In the verification, data to be input to the machine learning model was generated in each of the four patterns of Example 1, Comparative Example 1, Comparative Example 2, and Comparative Example 3 shown below, and the accuracy rate described above was calculated in each of Example 1 and Comparative Examples 1 to 3. The types of data (image data) input to the machine learning model were made different from each other in Example 1, Comparative Example 1, Comparative Example 2, and Comparative Example 3. FIG. 8 shows the types of data input to the machine learning model in the verification, and the accuracy rate of the output results from the machine learning model for the inspection results in the actually performed inspection, in each of Example 1, Comparative Example 1, Comparative Example 2, and Comparative Example 3.

[0088] As shown in Fig. 8, in Example 1, colored image data similar to the input image data in the above-mentioned embodiment was input to the machine learning model. In generating data for the colored images, three types of colored images were generated from the above-mentioned three types of captured images showing the state before reflow: a colored image of only the board, a colored image with only solder mounted on the board, and a colored image with solder and components mounted on the board. Then, the three types of colored images were arranged to generate colored image data.

[0089] In the colored image data, as in the above-mentioned embodiment, each of the components shown in the colored image is colored in a color corresponding to the color information. In fact, in the colored image data of Example 1, each of the components is colored in the same color as the example of input image data Da in FIG. 6. Therefore, in the colored image data, the pads are colored orange, the isolation parts are colored black, the resist is colored green, and the solder is colored gray. In the colored image data, the parts of the component are colored yellow in predetermined areas, and white in areas other than the predetermined areas.

[0090] In Comparative Example 1, line diagram data (line diagram image data) was input to a machine learning model. In generating the line diagram data, three types of line diagrams (line diagram images) were generated from the above-mentioned three types of captured images showing the state before reflow: a line diagram of only the board, a line diagram with only solder mounted on the board, and a line diagram with solder and components mounted on the board. Then, the three types of line diagrams were arranged to generate line diagram data. In the line diagram data, no color or pattern was added to the components shown in each of the line diagrams.

[0091] In Comparative Example 2, the grayscale image data was input to the machine learning model. In generating the grayscale image data, three types of grayscale images were generated from the above-mentioned three types of captured images showing the state before reflow: a grayscale image of only the board, a grayscale image with only solder mounted on the board, and a grayscale image with solder and components mounted on the board. Then, the grayscale image data was generated by arranging the three types of grayscale images. Since the grayscale image data was generated as described above, at least a part of the components in the grayscale images were colored in a color different from the color information.

[0092] In Comparative Example 3, the colored image data was input to the machine learning model. The colored image data was generated by arranging three types of colored images, namely, a colored image of only the board, a colored image of only solder mounted on the board, and a colored image of the board with solder and components mounted on it, in the same manner as in the colored image data in Example 1. However, in the colored image data in Comparative Example 3, the color of the resist and the color of the pad were exchanged from the colored image data in Example 1. That is, in the colored image data in Comparative Example 3, the pads were given green and the resist was given orange. Therefore, in the colored image data in Comparative Example 3, some of the components, i.e., the pads and the resist, were colored in a color different from the color information.

[0093] 8, the accuracy rate of the output results from the machine learning model for the test results in the actual test was 100% in Example 1. On the other hand, in Comparative Examples 1, 2, and 3, the accuracy rates were 60%, 80%, and 60%, respectively. Therefore, in all of Comparative Examples 1 to 3, the accuracy rate was 80% or less.

[0094] The above verification has demonstrated that by inputting image data in which each component is assigned a color corresponding to the color information into a machine learning model as image data based on an image before reflow, the machine learning model outputs inspection results after reflow with high accuracy. Therefore, it has been demonstrated that by inputting input image data generated as in the above-mentioned embodiment into a machine learning model, the inspection results after reflow when a product is manufactured based on the design in the design information can be determined with high accuracy.

[0095] According to at least one of these embodiments or examples, input image data showing a state before reflow is generated based on design information related to soldering, and a color or pattern corresponding to color information set for each type of component in the input image data is applied to each of multiple types of components. Then, using a machine learning model that inputs image data based on an image before reflow and outputs an inspection result in an inspection after reflow, the input image data is input to the machine learning model to determine the appropriateness of the design indicated by the design information. This makes it possible to provide an information processing device and an information processing method that enable the appropriateness of a design related to soldering to be determined with high accuracy at the design stage before manufacturing begins.

[0096] Although some embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are included in the scope and spirit of the invention, and are included in the scope of the invention and its equivalents described in the claims. [Explanation of symbols]

[0097] 1... manufacturing system, 2... solder mounting device, 3... component mounting device, 4... reflow device, 5... inspection device, 6... information processing device, 11-13... imaging device, 21... processing execution unit, 22... memory unit, 31... image generation unit, 32... judgment unit, 33... color information setting unit, 35... learning model generation unit, 36... cause identification unit, 40... machine learning model.

Claims

1. An information processing device relating to soldering of components to a substrate, comprising: an image generating unit that generates input image data showing a state before reflow based on design information related to soldering, and applies a color or pattern corresponding to color information set for each of a plurality of types of components shown in the input image data to each of the plurality of types of components; a determination unit that uses a machine learning model that receives image data based on an image before reflow and outputs an inspection result in an inspection after the reflow, and determines the appropriateness of a design indicated by the design information by inputting the input image data, in which a color or pattern corresponding to the color information is applied to each of the multiple types of components, into the machine learning model; An information processing device comprising:

2. 2. The information processing device of claim 1, further comprising a cause identification unit that, when the machine learning model outputs that the input of the input image data to the machine learning model will result in a defective inspection result after the reflow, identifies a cause of the defective inspection result in one or more of the input image data, the design information used to generate the input image data, and data generated in the process of generating the input image data from the design information.

3. 2. The information processing device of claim 1 , further comprising a color information setting unit that sets the color information for each of the multiple types of components such that the multiple types of components are distinguished from each other by the color information and each of the multiple types of components is specified by the color information.

4. the image generation unit generates external appearance elements of each of the plurality of types of components based at least on the design information related to soldering; The image generation unit applies a color or a pattern corresponding to the color information set for each of the plurality of types of components to the generated external element in generating the input image data. The information processing device according to claim 1 .

5. the image generation unit generates the input image data using design information about the base, design information about a metal mask used to mount solder on the base, and design information about the component, as the design information related to soldering; the image generation unit generates, based at least on the design information about the base, an outer appearance element of each of one or more types of components constituting the base, based at least on the design information about the metal mask, an outer appearance element of the solder, based at least on the design information about the metal mask, and based at least on the design information about the component, an outer appearance element of the multiple types of components, 5. The information processing device according to claim 4.

6. 6. The information processing device of claim 5, wherein the image generation unit generates the outline element of the solder based on, in addition to the design information about the metal mask, relationship data indicating a relationship between an opening in the metal mask and a shape of the solder after being mounted on the base.

7. 4. The information processing device according to claim 1, wherein the image generation unit generates a plurality of images from among three types of images, an image of only the base, an image with only solder mounted on the base, and an image with the solder and components mounted on the base, as images showing the state before the reflow in the input image data.

8. 4. The information processing device according to claim 1, further comprising a learning model generation unit that uses learning data in which image data based on an image showing a state before reflow and inspection results from an inspection actually performed after the reflow are associated to train a model through deep learning and generate the machine learning model.

9. 1. A method for processing information relating to soldering of a component to a substrate, comprising the steps of: generating input image data showing a state before reflow based on design information related to soldering, and applying a color or pattern corresponding to color information set for each of a plurality of types of components shown in the input image data to each of the plurality of types of components; using a machine learning model that receives image data based on an image before reflow and outputs an inspection result in an inspection after the reflow, inputting the input image data, in which a color or pattern corresponding to the color information is applied to each of the multiple types of components, into the machine learning model to determine the appropriateness of the design indicated by the design information; An information processing method comprising:

Images