Mounting substrate abnormality inference system and machine learning method
The abnormality estimation system addresses the challenge of low accuracy in mounted board assessments by using machine learning to update its inference model based on selective data set targeting, visualizing feature maps to correct bias and improve determination accuracy, thus reducing manual labor and enhancing reliability.
Patent Information
- Application Number
- PCT/JP2023/018993
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-22
- Publication Date
- 2024-11-28
AI Technical Summary
Existing systems for estimating abnormalities in mounted boards face challenges in achieving high determination accuracy, as they often rely on biased learning data and require manual secondary judgments, which can be labor-intensive and prone to errors.
An abnormality estimation system equipped with an inference model that uses machine learning to update parameters based on images and quality determination results, featuring a display unit for visualizing feature maps to selectively target data sets for learning, thereby correcting bias and improving accuracy.
The system achieves high determination accuracy by selectively updating the inference model with diverse data sets, reducing the burden on manual judgments and enhancing the reliability of secondary determinations.
Smart Images

Figure JP2023018993_28112024_PF_FP_ABST
Abstract
Description
Mounting board abnormality estimation system and machine learning method
[0001] The present invention relates to an anomaly inference system that infers an anomaly in a mounting board using an inference model, and a machine learning method for creating the inference model.
[0002] In circuit board production lines where electronic components are mounted on boards, visual inspection machines are installed to inspect whether the required components are mounted on the board in the correct condition after component mounting. The visual inspection machine automatically performs a primary judgment of the quality of the mounted board by capturing an image of the mounted board and performing required image processing on the acquired image. For mounted boards that are judged to be defective in this primary judgment, a worker visually judges the quality of the mounted board in a secondary judgment as the final judgment. A mounted board that is judged to be defective in the primary judgment may be judged to be non-defective in the secondary judgment.
[0003] Patent Literature 1 discloses an inspection system that performs machine learning on a data set of product images and pass / fail judgment results of the product to create an inference model, and then uses this inference model to judge the pass / fail of subsequent products. Introducing such an inference model-based judgment technology into the board production line to support the above-mentioned secondary judgment is desirable because it reduces the burden on workers. However, simply performing machine learning on a data set does not necessarily result in an inference model that makes highly accurate judgments.
[0004] Japanese Patent Application Laid-Open No. 2019-158684
[0005] An object of the present invention is to provide a mounting board anomaly estimation system equipped with an inference model with high judgment accuracy, and a machine learning method for creating the inference model.
[0006] A mounting board abnormality inference system according to one aspect of the present invention is a mounting board abnormality inference system having a display unit capable of displaying images, and includes: an imaging unit that images a mounting board on which components are mounted; an inspection unit that uses an inference model to infer an abnormality in the mounting board imaged by the imaging unit; a learning unit that updates parameters of the inference model through machine learning using a data set consisting of the image of the mounting board imaged by the imaging unit and a pass / fail determination result for the mounting board; and an information processing unit that extracts feature amount information of the mounting board based on the image of the mounting board, and displays a feature amount map that maps the feature amount information on the display unit.
[0007] A machine learning method according to another aspect of the present invention is a machine learning method for an inference model used to estimate an abnormality in a mounting board on which components are mounted, in which an inference model is created by machine learning using a dataset of images of a plurality of mounting boards and pass / fail determination results for the plurality of mounting boards, a selection is made as to whether or not a dataset of newly captured images of the mounting boards and the pass / fail determination results for the mounting boards is to be used as a subject for the machine learning, and when the dataset is newly selected as a subject for the machine learning, parameters of the inference model are updated using the existing dataset and the new dataset.
[0008] FIG. 1 is a block diagram showing a simplified configuration of a production line for mounting substrates to which the anomaly estimation system of the present invention is applied. FIG. 2 is a diagram showing a schematic configuration of the anomaly estimation system according to an embodiment of the present invention. FIG. 3 is a schematic diagram for explaining the creation of a feature map. FIGS. 4A to 4C are diagrams for explaining problems associated with creating an inference model. FIG. 5 is a schematic diagram for explaining the steps for creating an inference model according to this embodiment. FIGS. 6A to 6C are diagrams showing a process for selecting a dataset to be used for machine learning using a feature map. FIG. 7 is a block diagram showing the configuration of the anomaly estimation system according to this embodiment. FIG. 8 is a flowchart showing an example of a learning data selection process performed by the anomaly estimation system. FIG. 9 is a flowchart showing an example of a secondary judgment process performed by the anomaly estimation system. FIG. 10 is a diagram showing an example of an image display of an anomaly location.
[0009] Embodiments of the present invention will be described in detail below with reference to the drawings. An anomaly estimation system according to the present invention is a system that automatically determines the quality of a mounted board using an inference model created by machine learning using a data set consisting of multiple images of the mounted board on which components are mounted and quality determination results for the mounted board. A machine learning method according to the present invention is a method for creating the inference model. The mounted board is a product in which electronic components, such as chip components such as chip resistors and chip capacitors, ball bump components, and packaged components such as ICs, power supply-related components such as capacitors and transformers, connectors, heat sinks, etc., are mounted on a printed circuit board on which a circuit pattern is printed.
[0010] 1 is a block diagram showing a simplified configuration of a mounted board production line 1 to which the anomaly estimation system of the present invention is applied. The mounted board production line 1 is a line for mounting electronic components and the like onto printed circuit boards, and includes a board transport path TR that transports boards on which electronic components are mounted from right to left in Fig. 1, a plurality of work devices that are arranged on the board transport path TR and perform predetermined operations on the boards, and a line control device 10 that provides overall control of these work devices.
[0011] 1 shows the working devices arranged in tandem from upstream to downstream in the board transport direction: a printer 11, a print inspection device 12, a mounting device 13, a board inspection device 14, a reflow oven 15, and an appearance inspection device 16 (primary inspection device). At the upstream end of the mounted board production line 1 is a loader 17 that carries boards into the printer 11, and at the downstream end is an unloader 18 that removes produced boards from the appearance inspection device 16.
[0012] The printer 11 has a working section that applies solder to pads on a printed circuit board. For example, the printer 11 places a mask with openings where solder should be applied on the printed circuit board, and applies cream solder through the mask. The print inspection machine 12 takes an image of the printed circuit board with solder applied, determines the two-dimensional and / or three-dimensional shape of the solder, and inspects whether the position, amount, and height of the solder applied to the printed circuit board are appropriate.
[0013] The mounter 13 is equipped with a component mounting head and mounts required electronic components on the printed circuit board. The mounted circuit board production line 1 shown in FIG. 1 shows an example in which three mounters 13 are arranged in series. That is, from the upstream side, the first mounter 13A, the second mounter 13B, and the third mounter 13C are arranged in tandem. The board inspection machine 14 takes images of the printed circuit boards that have passed the mounters 13 and inspects the electronic components mounted on the printed circuit board for misalignment, lead misalignment, component lift, missing components, soldering defects, and the presence of foreign matter. The reflow furnace 15 heats the printed circuit board on which the electronic components have been mounted to melt the solder and fix the electronic components to the printed circuit board.
[0014] The visual inspection machine 16 takes an image of the printed circuit board after heat treatment in the reflow furnace 15, and similarly to the board inspection machine 14, inspects for misalignment of electronic components, lead misalignment, component lift, missing mounting, soldering defects, the presence of foreign matter, etc. In other words, it takes an image of the completed mounted board and automatically judges whether there is an abnormality in the mounted board based on that image. In this embodiment, an example is shown in which the anomaly estimation system according to the present invention is applied to secondary judgment as to whether a mounted board that has been automatically judged as a "defective product" by the visual inspection machine 16 is truly a "defective product" or whether it has over-judged a "defective product" despite being a "good product."
[0015] The line control device 10 controls the production work of mounted boards by comprehensively controlling the work devices that are provided on the mounted board production line 1, which are the printing machine 11, the printing inspection machine 12, the mounting machine 13, the board inspection machine 14, the reflow furnace 15, and the appearance inspection machine 16. The line control device 10 also collects various types of information output from the work devices. The various types of information include notification information such as status information that indicates the operating status of the work devices, error information that is output when a malfunction occurs in the operation, information requesting the replenishment of consumable materials such as electronic components, and information requesting some kind of action.
[0016] 2 is a diagram showing a schematic diagram of the overall configuration of the anomaly estimation system AES according to this embodiment. The anomaly estimation system AES includes the above-mentioned appearance inspection machine 16 having a camera 31 (imaging unit) that captures an image of the mounting board, an AI station 2 (inspection unit, learning unit, information processing unit), and a repair station 3.
[0017] The visual inspection machine 16 is a primary inspection device that makes a primary judgment as to whether the mounted boards produced on the mounted board production line 1 are good or bad, without using an inference model. A camera 31 captures an image of the mounted board carried into the visual inspection machine 16 along the board transport path TR, thereby acquiring an appearance image of the mounted board. The visual inspection machine 16 automatically judges whether the inspected mounted board satisfies the criteria by, for example, comparing a reference evaluation value that allows the above-mentioned inspection to be performed with an evaluation value extracted from the captured (acquired) image. Images of mounted boards that are judged as not satisfying the criteria and having an abnormality (hereinafter referred to as an NG judgment) as a result of the automatic judgment are sent to the AI station 2 and the repair station 3.
[0018] The repair station 3 is a work area where a secondary judgment is made, by visual inspection by an operator working on the mounted board production line 1, as to whether a mounted board that has been judged as NG in the primary judgment by the visual inspection machine 16 is pass or fail. The repair station 3 includes a judgment terminal 30 operated by the operator. An image of the mounted board that has been judged as NG is displayed on the monitor of the judgment terminal 30. The operator visually checks the image and makes a secondary judgment to finally determine whether the mounted board is a "defective" mounted board as judged in the primary judgment, or whether the primary judgment was an over-judgment and the mounted board should be judged as a "good" mounted board.
[0019] The AI station 2 accumulates images of mounting boards that have been judged as NG in the primary judgment by the visual inspection machine 16 (NG images), and performs machine learning using the NG images and the secondary judgment results for the mounting boards as a data set to construct an inference model for determining whether the mounting board is good or bad. As will be described in detail later, a feature of this embodiment is that the machine learning does not use all data sets as learning objects, but rather uses selected data sets as learning objects. The AI station 2 includes an AI processing device 21, an over-judged image DB 22, a selection terminal 23, and a learning DB 24.
[0020] The AI processing device 21 uses the inference model constructed by the machine learning to estimate abnormalities in a mounting board that is a new target for judgment. The over-judgment image DB 22 is a database that accumulates images of over-judged mounting boards that were judged as NG in the primary judgment but were judged as passable in the secondary judgment. Note that the over-judgment image DB 22 may also accumulate images of mounting boards that were judged as NG in the primary judgment and also judged as NG in the secondary judgment.
[0021] The selection terminal 23 is a terminal through which an operator selects a data set to be learned from among the data sets of images of mounting boards judged as over-judged and judgment results of non-defective products. The selection terminal 23 includes a display unit 7 capable of displaying images including information for the selection. In this embodiment, the information displayed on the display unit 7 is a feature amount map (described later with reference to FIG. 3 ) that maps feature amount information of the mounting board to be selected. The selection terminal 23 may also serve as the judgment terminal 30 of the repair station 3.
[0022] The learning DB 24 is a database that accumulates images of existing mounting boards that have been determined to be non-defective. Images of mounting boards that have been determined to be non-defective and that have been selected by the selection terminal 23 are also stored in the learning DB 24. A data set of the images of mounting boards stored in the learning DB 24 and the labeling results of the mounting boards being non-defective is used as the subject of machine learning to generate an inference model, and the inference model is updated as needed. Note that a data set of images of mounting boards that have been determined to be non-defective in the primary judgment, or images of mounting boards that have been determined to be non-defective in the primary judgment and the secondary judgment, and the labeling results of the mounting boards being "defective" may also be added as the subject of machine learning.
[0023] The AI station 2, which has been equipped with an inference model with high judgment accuracy through machine learning, functions as a secondary judgment device that supports the pass / fail judgment of the secondary judgment. It may be difficult to judge the pass / fail of a mounted board using visual inspection at the repair station 3. Furthermore, even if an experienced operator can make a pass / fail judgment, an inexperienced operator may have difficulty making the pass / fail judgment. In consideration of these factors, a pass / fail judgment request is sent from the repair station 3 to the AI station 2 for a mounted board that has been judged as NG in the primary judgment. Upon receiving this request, the AI station 2 performs an AI judgment of the pass / fail of the mounted board using the inference model obtained through the machine learning and inputs the judgment result to the repair station 3. The operator refers to the input judgment result and makes a final judgment on the pass / fail of the mounted board. The secondary judgment itself may be left to the AI station 2.
[0024] [Feature Amount Map] Next, the feature amount map displayed on the display unit 7 will be described. FIG. 3 is a schematic diagram for explaining general steps for creating a feature amount map. First, in step #1, a database DB1 is prepared, which accumulates a large number of data sets. The data sets are sets of images of mounting boards to be subjected to pass / fail judgment and pass / fail judgment results for those images. These data sets are the subject of machine learning for creating an inference model.
[0025] In the following step #2, feature information is calculated for each of the images. Examples of feature information obtained from the images include color features, edge features, texture features, and local features. Color features are features that use the RGB values of the image to express color information such as hue, saturation, and brightness. Edge features are features that express edges such as contours and boundary lines within the image. Texture features are features that express texture within the image. Local features are features that focus on a specific region within the image and express the pattern or shape within that specific region.
[0026] In step #3, an inference model is generated by machine learning of the data set in database DB1. The feature information calculated in step #2 is used to create the inference model. The inference model acts as a judgment formula for distinguishing between good and bad products.
[0027] In step #4, a feature map FM is created by mapping the feature values calculated in step #2. In other words, the feature values are visualized. When many different feature values are obtained and therefore there are many variables, it is difficult to represent all of the feature values in a two-dimensional or three-dimensional graph or the like for visualization. For this reason, a dimensionality reduction algorithm is applied when creating the feature map FM. Examples of dimensionality reduction algorithms that can be used include t-SNE, which compresses data to preserve local similarities, and UMAP, which compresses data to preserve the local structure of high-dimensional data.
[0028] In FIG. 3, a graph in which the feature quantities of good and defective products are plotted two-dimensionally is shown as an example of the feature quantity map FM. By using the inference model created in step #3, it is possible to display a good product distribution model that reflects the inference model in the feature quantity map FM. The good product distribution model surrounds the periphery of the feature quantity plots of good products. Note that FIG. 3 shows an example in which the feature quantities are plotted two-dimensionally for the sake of simplicity. Alternatively, the feature quantities may be plotted three-dimensionally, and a good product distribution model in which the feature quantity plots of good products are represented by a contour distribution may be displayed in the feature quantity map FM.
[0029] [Regarding the accuracy of inference models] In supervised learning in machine learning, it is desirable to have a large amount of training data and a variety of training data with different characteristics in order to create an accurate inference model. However, in reality, even if the number of training data is sufficient, it may be difficult to accumulate a variety of training data with different characteristics. In particular, for over-determined images of mounting boards, the primary judgment is often NG due to lead misalignment or foreign matter intrusion, and there is a tendency for highly similar data to be accumulated.
[0030] 4A to 4C are diagrams showing feature quantity maps, illustrating problems associated with creating an inference model. The feature quantity map in FIG. 4A displays a good-product feature quantity plot PGn and a defective-product feature quantity plot PN. A good-product distribution model M1 is also created to surround the good-product feature quantity plot PG. A criterion level LV for distinguishing between good and defective products is set based on the good-product distribution model M1.
[0031] The feature map in Figure 4(A) shows a bias in the distribution of non-defective product feature plots PG. Specifically, the feature map is composed of a first group PG1 consisting of non-defective product feature plots PG with similar features and a second group PG2 consisting of sporadic non-defective product feature plots PG scattered away from the first group PG1, resulting in a biased distribution of feature plots. An inference model created using training data with such biased features is likely to have low accuracy in pass / fail judgment. Furthermore, excessive training data in the first group PG1 can waste memory capacity for storing the training data. Therefore, a method of simply storing the entire dataset of images to be judged and performing machine learning to generate an inference model may not be desirable.
[0032] Methods for reducing bias in training data include undersampling and oversampling. FIG. 4B shows the undersampling method. Undersampling involves reducing the number of data items in the first group PG1 to match the number of data items in the second group PG2, which has a smaller number of data items. This adjustment creates a non-defective product distribution model M2 with a different range from the original non-defective product distribution model M1. When undersampling is employed, there is a concern that the accuracy of the inference model may decrease due to the reduced number of training data items.
[0033] 4C is a diagram showing an oversampling technique. In oversampling, adjustments are made to increase the number of data by copying data from the second group PG2 to match the first group PG1, which has a large number of data. This adjustment makes it easy to create a non-defective product distribution model M3 with a more limited range than the original non-defective product distribution model M1. When oversampling is employed, the second group PG2, which originally has a small number of data, is excessively trained, which raises concerns about a decrease in the accuracy of the inference model.
[0034] [Machine Learning Method According to the Embodiment] In this embodiment, a machine learning method capable of correcting bias in training data is used without relying on undersampling or oversampling. In short, an operator can arbitrarily select which data to add as training data for creating an inference model. To facilitate this selection, training data that are candidates for addition (in this embodiment, feature information of images of mounting boards) are displayed on a feature map. These points will be described with reference to FIGS. 5 and 6 .
[0035] 5 is a schematic diagram illustrating the steps for creating an inference model in this embodiment. It is assumed here that a certain amount of machine learning has been performed and an inference model already exists. Step A1 shows a feature map including a feature plot PG of a mounting board that has been determined to be a non-defective product based on the existing inference model. This feature map also shows the boundary of a non-defective product distribution model M that reflects the existing inference model and that covers the feature plot PG.
[0036] Step A2 shows a situation in which a new data set of inspection images IM for a mounting board has arrived. The inspection images IM are, for example, over-determined images that were judged as NG in the primary judgment but passed in the secondary judgment, as previously described with reference to FIG. 2. In step A2, feature amount information for the inspection images IM is calculated.
[0037] The next step A3 is a step in which it is determined whether to add the data set of the new inspection image IM to the machine learning target and update the existing inference model. The feature values of the inspection image IM calculated in step A2 are displayed in a feature value map as a sample feature value plot PT that is a candidate for adding to the machine learning target. By referring to this feature value map on the image, the operator can easily select whether to include the data set of the new inspection image IM in the machine learning target.
[0038] In making this selection, the operator can use the distance ds between the sample feature plot PT on the feature map and the point closest to the plot PT of the non-defective product distribution model M as an evaluation factor. To calculate the distance ds, if Gaussian-AD is used as the anomaly detection method, a multivariate Gaussian distribution or Mahalanobis distance can be used, and if PaDiM is used, a covariance matrix, a multivariate Gaussian distribution, random sampling, and Mahalanobis distance can be used. Of course, the distance ds can be calculated using methods other than those listed above.
[0039] Figures 6A to 6C illustrate the process of selecting a dataset to be used for machine learning using a feature map. Figure 6A shows a feature map similar to step A1 in Figure 5. To create this feature map, a dataset of non-defective products, i.e., images of existing printed circuit boards labeled as non-defective products, is first collected (step S1). Based on these images, various feature quantities as described above are calculated (step S2). After calculating the feature quantities, a process is performed to visualize them as a feature quantity map using the above-mentioned dimensionality reduction algorithm. Then, an inference model is set to define the range of non-defective products (step S3). Through the above process, a feature quantity map is created that includes a feature quantity plot PG for non-defective products and a non-defective product distribution model M that reflects the inference model.
[0040] FIG. 6B shows the process when a new data set of a mounting board determined to be a non-defective product is generated. First, images of the newly generated non-defective mounting boards are collected (step S4). As described in step A2 of FIG. 5, the additional image group in step S4 is, for example, a group of over-determined images. Next, feature quantities are calculated for each of the additional image groups (step S5). These feature quantities are processed using a dimensionality reduction algorithm and displayed as sample feature quantity plots PT1, PT2, and PT3 on the feature quantity map. In this embodiment, the additional non-defective product images are assumed to be images determined to be non-defective in the secondary determination, but they may also be images determined to be non-defective in the primary determination.
[0041] Here, it is desirable to limit the plots added to the feature map to those with low similarity to the existing good-product feature plot PG. In other words, it is desirable to display only the sample feature plots PT1 to PT3 with a relatively long distance ds (see step A3 in FIG. 5 ) from the good-product distribution model M on the feature map as candidates for additional learning. This is because increasing the number of plots with high similarity to the existing good-product feature plot PG, that is, plots that fit within the current good-product distribution model M, may only exacerbate the bias in the learning data. By displaying features with low similarity to the existing good-product feature plot PG, it is possible to bring the boundary of the good-product distribution model M, which distinguishes between good and defective products, closer to a more accurate decision boundary.
[0042] 6B, the operator selects an image of a mounting board to be used as an additional learning target from among the images corresponding to the sample feature plots PT1 to PT3 (step S7). Here, it is assumed that, of the sample feature plots PT1 to PT3, only the plot PT1 is selected as the learning target.
[0043] FIG. 6C shows the inference model update process. The inference model is recalculated using machine learning, including the dataset of images of the added mounting board (step S8). The feature map in FIG. 6C shows the state in which the sample feature plot PT1 selected in step S7 has been added as a new non-defective product feature plot PGA. A new non-defective product distribution model MA, which includes the non-defective product feature plot PGA, is also displayed based on the recalculated inference model. Steps S4 to S8 are then repeated, causing the non-defective product distribution model M to dynamically change to some extent and eventually converge.
[0044] [Specific Example of Abnormality Estimation System] Next, a specific example of the configuration of an abnormality estimation system will be illustrated. FIG. 7 is a block diagram showing the configuration of the abnormality estimation system AES of this embodiment. The abnormality estimation system AES includes a camera 31 (imaging unit), a learning unit 4, a memory unit 5, an information processing unit 6, and a display unit 7. As illustrated in FIG. 2 , the camera 31 is provided in the visual inspection machine 16 and is a device that captures an image of a mounting board 32 on which components have been mounted. The learning unit 4, the memory unit 5, and the information processing unit 6 are functional units provided in the AI station 2 shown in FIG. 2. The learning unit 4 and the information processing unit 6 are configured by a processor capable of information processing. The memory unit 5 is configured by a memory area, a hard disk drive, or the like that can be directly accessed by the processor. The display unit 7 is configured by a liquid crystal display or the like that can display images, and is, for example, a monitor provided in the selection terminal 23 shown in FIG. 2.
[0045] The learning unit 4 has a function of performing machine learning on a dataset of images of the mounting board 32 captured by the camera 31, and performing AI judgment on the quality of the mounting board 32 using an inference model obtained by the machine learning. Functionally, the learning unit 4 includes an input receiving unit 40 and a learning execution unit 43. The input receiving unit 40 is a functional unit that receives input operations from an operator OP viewing the display unit 7, and includes a selection unit 41 and an information addition unit 42.
[0046] The selection unit 41 accepts a selection of whether to use a dataset of images of the mounting board 32 captured by the camera 31 and the pass / fail judgment results of the mounting board 32 as the subject of machine learning. This selection corresponds to the selection shown in step S7 of FIG. 7 . The selection unit 41 does not need to select a dataset at the beginning of machine learning. The selection unit 41 may be used to accept a selection of whether to use newly captured images of the mounting board 32 captured by the camera 31 as the subject of machine learning for updating the inference model after an inference model has been created by accumulating a predetermined number of datasets. The information addition unit 42 accepts labeling of the pass / fail judgment for the captured mounting board 32. The dataset of the images selected by the selection unit 41 and the labeling of "good" or "defective" added by the information addition unit 42 is the subject of machine learning. For example, the dataset of the over-judged images and the labeling of "good" as described above, as well as a dataset of images judged as "NG" in both the primary and secondary judgments and the labeling of "defective" can be the subject of machine learning.
[0047] The learning execution unit 43 uses machine learning of the data set to create an inference model that judges the quality of the mounting board 32. After the inference model has been created, the learning execution unit 43 performs processing to update the existing inference model when a data set for a mounting board 32 newly selected by the operator OP is generated in the selection unit 41. In other words, the learning execution unit 43 updates the parameters of the current inference model using a group of existing data sets and a new data set.
[0048] The storage unit 5 is a functional unit that stores various types of information related to the anomaly estimation system AES, and includes an image memory 51, a data set storage unit 52, a model information storage unit 53, and an inspection result storage unit 54. The storage unit 5 includes storage units corresponding to the overdetermined image DB 22 and the learning DB 24 illustrated in FIG. 2. The image memory 51 stores image data of the mounting board 32 captured by the camera 31. When the operator OP selects an image in the selection unit 41, the image data of the mounting board 32 is read from the image memory 51.
[0049] The dataset storage unit 52 stores a dataset of images of the mounting board 32 and pass / fail determination results for the mounting board 32. After the selection unit 41 starts selecting a dataset, only the selected dataset is stored in the dataset storage unit 52. The learning execution unit 43 performs machine learning using the dataset stored in the dataset storage unit 52, and creates and updates an inference model.
[0050] The model information storage unit 53 stores information constituting the inference model created by the learning execution unit 43. When the parameters of the inference model are updated by machine learning of a new data set, the learning execution unit 43 updates the information of the existing inference model stored in the model information storage unit 53. The inspection result storage unit 54 stores the inspection results of the mounting board 32 by the inspection unit 63, which will be described later.
[0051] The information processing unit 6 functionally includes a display control unit 61, a feature calculation unit 62, and an inspection unit 63. The display control unit 61 displays predetermined information on the display unit 7. The information includes various information stored in the storage unit 5, a feature map in which the feature information calculated by the feature calculation unit 62 is mapped, and inspection results from the inspection unit 63.
[0052] A specific example of the display mode of the feature map by the display control unit 61 is as shown in FIG. 6 . That is, as shown in FIG. 6B , a non-defective product feature plot PG, which is feature information for the trained mounting board 32, and sample feature plots PT1, PT2, and PT3 are displayed on the same feature map. The sample feature plots PT1 to PT3 are feature information for a non-defective mounting board 32 that was newly captured by, for example, the camera 31 of the visual inspection machine 16 and was judged to be an over-judgment in the secondary judgment. In this way, by displaying the non-defective product feature plot PG and the sample feature plots PT1 to PT3 on the same feature map, the relationship between the trained feature information and the new feature information is visualized. This provides easy-to-understand information when the operator OP decides whether to subject a new data set of the mounting board 32 to machine learning.
[0053] The display control unit 61 also displays the group of non-defective product feature plots PG on the feature map, along with the boundary lines of the non-defective product distribution model M. The non-defective product distribution model M is area information on the feature map that is created based on the distribution of the non-defective product feature plots PG, which are trained feature information. In this way, the display control unit 61 not only displays the non-defective product feature plots PG scattered on the feature map, but also displays them along with the area information of the non-defective product distribution model M. This display makes it possible to visualize the relationship between the sample feature plots PT1 to PT3, which are candidates for learning, and the non-defective product distribution model M, which indicates the range currently defined as a non-defective product. Therefore, it is possible to provide the operator OP with more easily understandable information when determining whether to subject a new dataset to machine learning.
[0054] The feature amount calculation unit 62 performs processing to extract feature amount information of the mounting board 32 based on the image of the mounting board 32. The feature amount calculation unit 62 performs predetermined image processing and the like on the images of the mounting board 32 acquired by the camera 31 and stored in the image memory 51 of the storage unit 5, and derives feature amount information of each image. As described above, the feature amount information obtained from the image includes, for example, color features, edge features, texture features, and local features.
[0055] When the accuracy of the inference model reaches a required level, the inspection unit 63 performs AI assistance in the secondary judgment of the mounting board 32 that has been judged as NG by the visual inspection machine 16. When the inspection unit 63 receives an AI judgment request for the secondary judgment from the operator OP, it uses the inference model to perform an abnormality estimation process for the mounting board 32 that is the target of the secondary judgment. Regardless of the AI judgment request, the inspection unit 63 may be caused to perform the abnormality estimation process for all mounting boards 32 that are the target of the secondary judgment. The inspection unit 63 includes an estimation unit 64, a judgment unit 65, and an abnormality data calculation unit 66.
[0056] The estimation unit 64 uses the inference model stored in the model information storage unit 53 to calculate an evaluation value for the image of the mounting board 32 to be evaluated, for example in the form of a determinant. The inference model in the model information storage unit 53 is updated as needed by the learning execution unit 43, so the estimation unit 64 always uses the latest inference model to evaluate the image of the mounting board 32. The determination unit 65 uses the evaluation result calculated by the estimation unit 64 to determine whether the mounting board 32 is a non-defective product. The determination result of the determination unit 65 is displayed as an inspection result on the display unit 7, which is visually confirmed by the operator OP.
[0057] The abnormality data calculation unit 66 performs a process of converting abnormal locations of the mounting board 32 that have been determined to have an abnormality by abnormality estimation using an inference model into data. The display control unit 61 converts the converted abnormal locations into images and displays them on the display unit 7. The abnormality data calculation unit 66 also derives an evaluation value related to the degree of abnormality of the abnormal locations. The display control unit 61 displays the evaluation value on the display unit 7. This point will be described later with reference to FIG. 10 .
[0058] 8 is a flowchart showing an example of a learning data selection process performed by the anomaly estimation system AES. First, the appearance inspection machine 16 performs an automatic inspection as a primary judgment on the mounting boards 32 produced on the mounting board production line 1 (step S11). As described above, the appearance inspection machine 16 judges whether the mounting boards 32 are good or bad based on the image of the mounting boards 32 captured by the camera 31.
[0059] Next, it is confirmed whether the result of the primary judgment by the visual inspection machine 16 is a pass or fail judgment (step S12). If the result is a pass judgment, the feature calculation unit 62 calculates feature information based on the image of the mounting board 32 (step S13). Next, the display control unit 61 performs a dimensionality reduction process on the calculated feature information, converts the feature information into a plot, and displays it superimposed on the feature map displayed on the display unit 7 (step S14).
[0060] The operator OP, who has visually confirmed the feature map displayed on the display unit 7, uses the selection unit 41 to select whether or not to add the data set of the image of the mounting board 32 and the OK judgment result as learning data for machine learning in the learning unit 4 (step S15). When the selection unit 41 receives an input of "Subject to machine learning" (YES in step S15), the data set is stored in the data set storage unit 52 (step S16).
[0061] Thereafter, the added dataset is subjected to machine learning by the learning execution unit 43. The learning execution unit 43 adds the additional dataset, performs a learning process, and updates the parameters of the inference model (step S17). Information on the updated inference model is stored in the model information storage unit 53. On the other hand, if the selection unit 41 receives an input indicating that the dataset is "not to be subjected to machine learning" (NO in step S15), steps S16 and S17 are skipped and the process ends.
[0062] On the other hand, if the result of the primary judgment in step S12 is a negative judgment, the operator OP or the inspection unit 63 performs a secondary judgment to make a final judgment on the quality of the mounting board 32 (step S18). The result of the secondary judgment is then confirmed (step S19). If the secondary judgment is a positive judgment, that is, if the primary judgment was an over-judgment, the process proceeds to step S13, where the above-mentioned processing is executed. That is, a routine is executed to determine whether or not to add the image of the mounting board 32 that was the over-judgment and the data set of the judgment result as a new learning target. On the other hand, if the result of the secondary judgment is also a negative judgment, the processing ends.
[0063] 9 is a flowchart showing an example of the secondary judgment process of step S18 shown in FIG. 8. When an AI assistance request is received from an operator OP, an inference model to be used in the estimation unit 64 of the inspection unit 63 is first selected (step S21). The AI assistance request is received by the information processing unit 6 when an inspection item that is difficult for the operator OP to judge pass / fail by visual inspection, such as a soldering defect of "unsoldered," needs to be judged. In this case, in step S21, an inference model suitable for estimating the above-mentioned "unsoldered" anomaly is selected. In this way, it is desirable to prepare an appropriate inference model for each pass / fail judgment category through machine learning.
[0064] Next, the estimation unit 64 cuts out an image area for which pass / fail judgment is to be performed using the inference model from the entire image of the mounting board 32 that is the target of the secondary judgment (step S22). This process is a trimming of the target area for pass / fail judgment, and is a process for removing image areas unnecessary for judgment, thereby shortening processing time and improving judgment accuracy. Thereafter, the estimation unit 64 applies the inference model to the image cut out in step S22 to perform an abnormality estimation process. Furthermore, the abnormality data calculation unit 66 calculates an evaluation value for the degree of abnormality of the area to be judged for abnormality, i.e., the degree of abnormality, based on the application result of the inference model (step S23).
[0065] Next, the operator OP selects whether or not to perform AI automatic judgment, or confirms whether or not the setting for adopting AI automatic judgment is set (step S24). If AI automatic judgment is to be performed (YES in step S24), the judgment result of the inspection unit 63 becomes the final judgment. In this case, it is confirmed whether or not the abnormality degree calculated by the abnormality data calculation unit 66 is equal to or less than a predetermined threshold (step S25). If the abnormality degree is equal to or less than the threshold (YES in step S25), the judgment unit 65 issues an "OK judgment" for the mounting board 32 (step S26). If the abnormality degree exceeds the threshold (NO in step S25), the judgment unit 65 issues an "NG judgment" for the mounting board 32 (step S30).
[0066] On the other hand, if AI automatic judgment is not performed (NO in step S24), that is, if the operator OP makes a final judgment visually by referring to the results derived by the inspection unit 63, the display control unit 61 displays the degree of abnormality calculated by the abnormality data calculation unit 66 on the display unit 7 along with an image of the judgment location (step S27). Fig. 10 is a diagram showing an example of an image display of the abnormality judgment location. An abnormality display image 8 is displayed on the display unit 7. The abnormality display image 8 is an image showing a part of the mounting board 32 where the operator OP will make a pass / fail judgment, such as a soldered portion of a lead of an electronic component. A focus area 81 for abnormality judgment is additionally displayed in a dotted frame on the abnormality display image 8.
[0067] The display unit 7 further displays an abnormality evaluation value display section 82 and an abnormality degree display section 83. The abnormality evaluation value display section 82 is a display field for the abnormality evaluation value calculated by the abnormality data calculation unit 66, and displays a numerical value indicating the abnormality degree of the mounting board 32 being judged. The abnormality degree display section 83 is a display field that visually indicates the abnormality degree. For example, the abnormality degree display section 83 displays a color indicator that expresses the abnormality degree using a single color shade or a color change, and the display control unit 61 superimposes the color indicator corresponding to the abnormality degree on the board image in the focus area 81. By displaying the display as shown in FIG. 10 , the operator OP can simultaneously view an image of the abnormality judgment location on the mounting board 32 and the evaluation value of the abnormality degree. This makes it easier for the operator OP to grasp the degree of abnormality of the mounting board 32 being judged as being the secondary judgment target.
[0068] 9 , the operator OP visually performs a secondary judgment by referring to the display in step S27. The result of the secondary judgment is received by the judgment unit 65 of the inspection unit 63 (step S28). Then, it is confirmed whether the judgment result is an NG judgment (step S29). If the operator OP inputs an NG judgment (YES in step S29), the NG judgment is confirmed (step S30). If the judgment is an OK judgment (NO in step S29), the OK judgment is confirmed (step S26).
[0069] As described above, the anomaly estimation system AES of this embodiment allows for the selection of which data sets to use in machine learning, rather than subjecting all of the data sets of images of the mounting board 32 and their pass / fail judgment results to machine learning. This selection is facilitated by the display unit 7 displaying a feature map of feature information for the mounting board 32, facilitating the operator OP's selection process. This allows for the operator OP to exclude new data sets that are highly similar to numerous existing data sets from machine learning, while actively selecting data sets with evaluations that are close to the borderline of defective products as the target of machine learning. Therefore, various data sets with different features can be accurately trained to update the inference model, providing a mounting board anomaly estimation system equipped with an inference model with high judgment accuracy.
[0070] The present invention is not limited to the above-described embodiment, and various modifications are possible. For example, in the above-described embodiment, the visual inspection machine 16 performs an abnormality determination based on an evaluation extracted from a captured image of a mounting board without using an inference model. Alternatively, the visual inspection machine 16 may also perform an abnormality determination using an inference model.
[0071] [Inventions Included in the Above-Described Embodiments] The above-described embodiments include inventions having the following configurations.
[0072] A mounting board abnormality inference system according to one aspect of the present invention is a mounting board abnormality inference system having a display unit capable of displaying images, and includes: an imaging unit that images a mounting board on which components are mounted; an inspection unit that uses an inference model to infer an abnormality in the mounting board imaged by the imaging unit; a learning unit that updates parameters of the inference model through machine learning using a data set consisting of the image of the mounting board imaged by the imaging unit and a pass / fail determination result for the mounting board; and an information processing unit that extracts feature amount information of the mounting board based on the image of the mounting board, and displays a feature amount map that maps the feature amount information on the display unit.
[0073] According to the above-described mounting board anomaly estimation system, an information processing unit causes a feature map, which maps feature information of the mounting board, to be displayed on a display unit. By referring to the feature map, an operator can easily select which mounting board datasets to use in machine learning. For example, a new dataset that is highly similar to numerous existing datasets can be excluded from machine learning, while a dataset with an evaluation close to the borderline of defective products can be actively used in machine learning. This allows various datasets with different characteristics to be accurately trained, allowing an inference model to be constructed or updated. Therefore, a mounting board anomaly estimation system equipped with an inference model with high judgment accuracy can be provided.
[0074] In the above-described anomaly estimation system, it is desirable that the information processing unit displays, on the same feature map, feature information about the learned mounting board and feature information about the mounting board newly imaged by the imaging unit.
[0075] According to this aspect, the trained feature information and the feature information of a new mounting board that is a candidate for learning are displayed on the same feature map. That is, the relationship between the trained feature information and the new feature information is visualized. Therefore, it is possible to provide easy-to-understand information when an operator decides whether or not to subject a dataset of a new mounting board to machine learning.
[0076] In the above-described anomaly estimation system, it is desirable that the information processing unit displays, on the feature map, the feature information about the learned mounting board, together with region information created based on the distribution of feature information about the non-defective mounting board.
[0077] According to this aspect, feature information about the learned mounting board is not displayed scatteredly on the feature map, but is displayed together with information about the region of good products. This makes it possible to visualize the relationship between feature information that is a candidate for learning and the range currently defined as good products. Therefore, it is possible to provide the operator with more understandable information when deciding whether to subject a new dataset to machine learning.
[0078] In the above-described anomaly estimation system, the learning unit may include a selection unit that receives a selection as to whether or not to subject a data set of an image of a mounting board newly captured by the imaging unit and a pass / fail determination result for the mounting board to machine learning.
[0079] According to this aspect, the selection of a dataset is accepted from an operator through the selection unit, thereby facilitating the operation of selecting a dataset to be used for machine learning.
[0080] In the above-mentioned anomaly estimation system, the learning unit may be provided with a learning execution unit that updates the parameters of the inference model using an existing dataset and the new dataset when the dataset is newly selected as a target for machine learning.
[0081] According to this aspect, it is possible to continually update the inference model to one with higher determination accuracy.
[0082] In the above-mentioned anomaly estimation system, it is desirable to include a memory unit that stores information about the inference model, and the learning unit updates the information about the inference model in the memory unit when parameters of the inference model are updated, and the inspection unit uses the updated inference model to estimate an abnormality in the mounting board.
[0083] According to this aspect, it is possible to always use the latest inference model to estimate abnormalities in the mounting board, thereby enabling the accuracy of determining whether the mounting board is good or bad to be gradually improved.
[0084] The above-described anomaly estimation system may further include an anomaly data calculation unit that converts an abnormal location of a mounting board that has been determined to have an abnormality by anomaly estimation using the inference model into data, and the information processing unit may further include a display control unit that converts the converted data into an image of the abnormal location and displays it on the display unit.
[0085] According to this aspect, the abnormal portion is imaged and displayed on the display unit, making it easier for the operator to grasp the nature of the abnormal portion.
[0086] In the above-described abnormality estimation system, it is preferable that the abnormality data calculation unit derives an evaluation value relating to the degree of abnormality of the digitized abnormal location, and the display control unit displays the evaluation value on the display unit.
[0087] According to this aspect, an abnormality location on the mounting board is displayed together with an evaluation value of the degree of abnormality, making it easier for the operator to grasp the degree of the abnormality.
[0088] The above-described abnormality estimation system preferably further comprises a primary inspection device that makes a primary determination of an abnormality in the mounting board based on the image of the mounting board captured by the imaging unit without using the inference model, and the inspection unit preferably makes a secondary determination of a mounting board that has been determined to have an abnormality by the primary determination, using the inference model to estimate an abnormality in the mounting board.
[0089] According to this aspect, the secondary judgment that was previously performed by an operator through visual inspection can be automated by estimating abnormalities using an inference model, thereby reducing the burden on the operator.
[0090] A machine learning method according to another aspect of the present invention is a machine learning method for an inference model used to estimate an abnormality in a mounting board on which components are mounted, in which an inference model is created by machine learning using a dataset of images of a plurality of mounting boards and pass / fail determination results for the plurality of mounting boards, a selection is made as to whether or not a dataset of newly captured images of the mounting boards and the pass / fail determination results for the mounting boards is to be used as a subject for the machine learning, and when the dataset is newly selected as a subject for the machine learning, parameters of the inference model are updated using the existing dataset and the new dataset.
[0091] This machine learning method does not subject all datasets to machine learning, but includes a process of selecting which datasets to subject to machine learning. For example, it is possible to exclude new datasets that are highly similar to many existing datasets from machine learning, while actively subjecting datasets with evaluations that are close to the borderline of defective products to machine learning. This allows the inference model to be updated by accurately learning various datasets with different characteristics. Therefore, it is possible to create an inference model with high judgment accuracy.
[0092] According to the present invention as described above, it is possible to provide a mounting board anomaly estimation system equipped with an inference model with high judgment accuracy, and a machine learning method for creating the inference model.
Claims
1. A mounting board anomaly inference system having a display unit capable of displaying images, comprising: an imaging unit that images a mounting board on which components are mounted; an inspection unit that uses an inference model to infer an abnormality of the mounting board imaged by the imaging unit; a learning unit that updates parameters of the inference model through machine learning using a data set of the image of the mounting board imaged by the imaging unit and a pass / fail determination result for the mounting board; and an information processing unit that extracts feature amount information of the mounting board based on the image of the mounting board, and causes a feature amount map that maps the feature amount information to be displayed on the display unit.
2. A mounting board anomaly estimating system as claimed in claim 1, wherein the information processing unit displays, on the same feature map, feature information about the learned mounting board and feature information about a mounting board newly imaged by the imaging unit.
3. A mounting board anomaly estimating system as claimed in claim 2, wherein the information processing unit displays, on the feature map, feature information about the learned mounting board, together with area information created based on the distribution of feature information about the mounting boards that are good.
4. A system for estimating an anomaly in a mounting board according to claim 2, wherein the learning unit includes a selection unit that receives a selection as to whether or not a data set consisting of an image of the mounting board newly captured by the imaging unit and a pass / fail determination result for the mounting board is to be subject to machine learning.
5. A system for estimating an anomaly in a mounting board as described in claim 3, wherein the learning unit includes a learning execution unit that updates parameters of the inference model using an existing dataset and the new dataset when the dataset is newly selected as a target for machine learning.
6. A mounting board anomaly inference system according to any one of claims 1 to 5, further comprising: a memory unit for storing information about the inference model; the learning unit updates the information about the inference model in the memory unit when a parameter of the inference model is updated; and the inspection unit uses the updated inference model to infer an abnormality in the mounting board.
7. A system for estimating anomaly in a mounting board according to any one of claims 1 to 5, wherein the inspection unit includes an abnormality data calculation unit that converts an abnormal location in the mounting board that has been determined to have an abnormality by anomaly estimation using the inference model into digital data, and the information processing unit includes a display control unit that converts the converted digital data into an image of the abnormal location and displays it on the display unit.
8. A system for estimating an abnormality in a mounting board as described in claim 7, wherein the abnormality data calculation unit derives an evaluation value relating to the degree of abnormality of the digitized abnormality location, and the display control unit causes the display unit to display the evaluation value.
9. The mounting board anomaly estimation system according to claim 1, further comprising a primary inspection device that performs a primary judgment of an abnormality in the mounting board based on an image of the mounting board captured by the imaging unit without using the inference model, and the inspection unit performs a secondary judgment, using the inference model, to estimate an abnormality in a mounting board that has been determined to have an abnormality by the primary judgment.
10. A machine learning method for an inference model used to estimate an abnormality in a mounting board on which components are mounted, comprising the steps of: creating an inference model by machine learning using a dataset of images of a plurality of mounting boards and pass / fail determination results for the plurality of mounting boards; selecting whether or not to use a dataset of newly captured images of the mounting board and the pass / fail determination results for the mounting board as a subject of the machine learning; and, when the dataset is newly selected as a subject of the machine learning, updating parameters of the inference model using the existing dataset and the new dataset.
Citation Information
Patent Citations
Method and device for sorting defective image and manufacturing method of semiconductor device using them
JP2001156135A
Mounting substrate visual inspection method
JP2008185395A
Image processing device, image processing method, and image processing program
JP2021117152A
Component inspection device
JP2022108855A
Evaluation program, evaluation method, and information processor
JP2022182702A
