Inspection of printed circuit board assemblies
By introducing a machine learning model as a classifier into the AOI system, based on numerical measurement results and metadata, the problem of high false alarm rate in printed circuit board assemblies was solved, achieving more efficient automated inspection and production process optimization.
Patent Information
- Application Number
- CN202380032109.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-03-30
- Filing Date
- 2023-02-28
- Publication Date
- 2024-11-26
AI Technical Summary
Existing automated optical inspection systems have a large number of false alarms in printed circuit board assemblies (PCBAs), which increases the workload of manual inspection, reduces production efficiency, and makes it difficult to adapt to the needs of different inspection types.
Using a machine learning model as a classifier, based on the numerical measurement results and metadata of the AOI system, the false alarm rate is reduced through feature vectors and dimensionality reduction techniques. It provides a binary output to distinguish between real and false errors and is integrated into the production process to control the conveyor belt movement.
It reduces the workload of manual inspection, increases the first pass rate, lowers production costs, improves production efficiency, and adapts to the needs of different inspection types.
Smart Images

Figure CN119032378A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to inspection of printed circuit board assemblies (PCBAs). In addition, the present disclosure relates to techniques for verifying inspection results of an automated optical inspection system. Background Art
[0002] The manufacture of printed circuit board assemblies can be based on different mounting techniques for components on printed circuit boards. First, components such as resistors or diodes are automatically placed on the surface of a printed circuit board using the so-called surface mounting technology. Surface mounting technology can be applied to small components and integrated circuits (ICs). Secondly, through-hole technology (THT) can be used. THT is suitable for inserting components with pins or wires that must be mounted on a printed circuit board by passing through holes in the printed circuit board. The pin part must be soldered on the other side of the printed circuit board. THT is typically applied to RGB assemblies that contain large components to be assembled, such as capacitors and / or coils.
[0003] Once the components are soldered in place, i.e. after the soldering process, the printed circuit board assembly PCBA needs to be tested. Often, movement during the soldering process will result in poor quality connections or a complete lack of connection. Short circuits are also a side effect of this movement, as misplaced components can sometimes connect parts of the circuit that should not be connected. Therefore, inspection of the produced PCBA is necessary.
[0004] Automated Optical Inspection (AOI) is an inspection method for PCBAs. An AOI system uses one or more cameras to inspect the PCBA. The cameras can be arranged at different angles to view the solder joints. Solder joints of varying quality reflect light in different ways, allowing the AOI system to identify lower quality solder. AOI systems do this at very high speeds, allowing the AOI system to process a large number of PCBAs in a relatively short period of time. Even when using multiple cameras or viewing angles, the inspection results of the AOI system may still be incorrect, and false positives, also known as pseudo-errors or false positives, may be obtained as results from the AOI system.
[0005] From European patent application publication EP 3941177 A1 a method for inspecting a printed circuit board (PCB) assembly is known. In this document it is proposed to find a comparison between an object-based analysis and stored assembly information of the PCB in order to identify missing components.
[0006] According to European patent application with application number 20216995.9, it is known to verify inspection results of an AOI system by using a machine learning model for classification, thereby determining false errors of an AOI system.
[0007] From the publication Thielen N., Werner D., Schmidt K., Seidel R., Reinhardt A., Franke J.: A Machining Learning Based Approach to Detect False Calls in SMT Manufacturing, 43rd International Spring Seminar on Electronics Technology, ISSE 2020 (Demanovska Valley, 14, Mai 2020-15. Mai 2020), In: Proceedings of the International Spring Seminar on Electronics Technology 2020, DOI: 10.1109 / ISSE49702.2020.9121044, it is known to use numerical measurement data generated by AOI systems for model training, testing and validation. Among them, the feasibility of using subsequent machine learning models to identify false alarms of AOI systems is studied. However, the problem still exists. Summary of the invention
[0008] For example, in order to verify a PCBA for which an AOI system has determined a defect, an operator needs to manually inspect the PCBA. Therefore, this manual verification requires additional effort. Therefore, the aim is to reduce the effort required for manual inspection by reducing the number of false positives (also called false alarms or pseudo errors) determined by the AOI system. A further aim is to perform this verification of the inspection results of the AOI system (even in the case of using different inspection types, i.e. when the AOI system allows configuration of various inspection types).
[0009] This object is achieved by the subject matter defined in the independent claims 1 , 10 to 14 , respectively. BRIEF DESCRIPTION OF THE DRAWINGS
[0010] Figure 1 A schematic diagram of a production line for producing and inspecting printed circuit board assemblies is shown.
[0011] Figure 2 A schematic diagram showing steps performed based on the inspection results of AOI.
[0012] Figure 3 A schematic diagram showing the verification of the inspection results by the classifier.
[0013] Figure 4A schematic diagram showing an AOI system and a device communicatively connected to the AOI system, wherein the device is used to verify an inspection result of the AOI system.
[0014] Figure 5 A schematic diagram of the feature vector is shown.
[0015] Figure 6 A schematic diagram of the classifier is shown.
[0016] Figure 7 Schematic diagram of a confusion matrix showing the misclassifications of a classifier.
[0017] Figure 8 A schematic diagram showing exemplary steps according to an embodiment is shown.
[0018] Fig. 9 A schematic diagram showing exemplary steps according to the first embodiment.
[0019] Fig.10 A schematic diagram showing exemplary steps according to the second embodiment.
[0020] Fig.11 A schematic diagram showing exemplary steps according to a third embodiment.
[0021] Fig.12 A schematic diagram showing exemplary steps according to a fourth embodiment.
[0022] Fig.13 A schematic diagram showing exemplary steps according to a fifth embodiment.
[0023] Fig.14 A schematic diagram of a device including an interface as well as a processor and a memory is shown.
[0024] Fig.15 A schematic diagram showing exemplary steps according to a sixth embodiment. DETAILED DESCRIPTION
[0025] In electronic product production (e.g., production line 100), electronic components are electrically and mechanically connected to a printed circuit board (PCB), for example, using through-hole technology (THT) or surface mount technology (SMT), such as Figure 1 As shown. In most cases, the electronic components are mounted on the printed circuit board at a placement station 11b (such as a manual placement station or with the aid of a placement robot, as the case may be). The printed circuit board assembly (PCBA) can be transferred to a conveyor belt. The conveyor belt moves the PCBA along the production (production line), for example, through a soldering station and a subsequent inspection station 10, etc.
[0026] Before the placement station 11b, solder paste can be applied to the printed circuit board by solder paste printing, for example, in the case of using SMT. Thereafter, the solder paste applied to the printed circuit board can be checked at the solder paste inspection station 11a. Then, the printed circuit board is driven into the soldering station 12 together with the mounted electronic components, where tin creates a conductive connection between the circuit board and the electronic components. A reflow oven or a wave soldering machine can be used to perform the soldering process. After the soldering process, the contact points or solder joints of the PCBA and their features are checked at the inspection station 10 for quality control. For this purpose, an automatic optical inspection AOI system can be used. The AOI system includes hardware components (such as digital sensors, computers, etc.) for image processing. The possible defects that the AOI system at the inspection station 10 should detect are missing components, bent components, solder joints with insufficient wetting (cold or open solder joints), or tin blue (i.e., electrical contacts of solder joints that should have an electric potential). For this reason, one or more images of the PCBA are captured and image processing of the one or more images is performed. For this reason, numerical measurements are generated by the AOI system based on image processing. These numerical measurements can be compared with corresponding thresholds to determine one or more defects or errors of the PCBA. If the image processing detects one or more errors on the PCBA, the PCBA is sent to the diagnostic station 13 via a return route for manual inspection. The on-site operator then performs a manual (visual) inspection, for example, via a display D on the diagnostic dashboard. Therefore, typically, the production of a PCBA requires one or more (i.e., multiple) production steps. The quality of the PCBA is determined by these production steps and verified by inspection.
[0027] According to the inspection results, the faulty PCBA can be repaired or scrapped. Data and control commands can be transmitted between various production stations via the data bus 101. To this end, the production stations are communicatively connected to the data bus 101.
[0028] After the soldering step, the PCBA production process ends, and the final inspection will test the functionality of the PCBA. This inspection is called a "functional test." This test simulates the normal conditions in which the PCBA will operate. Electric power and analog signals are run through the PCBA in this test while the tester monitors the electrical characteristics of the PCBA. If any of these characteristics (including voltage, current, or signal output) show unacceptable fluctuations or peaks outside the predetermined range, the PCBA fails the test. The failed PCBA can then be recycled or scrapped, depending on the company's standards.
[0029] The AOI system 10 is operable to provide an inspection result of a printed circuit board assembly 1. The inspection result preferably indicates whether the PCB assembly meets the quality requirements. The inspection result I1 can preferably be represented by "0" or "1" (in binary representation). For example, "0" indicates that the individual PCB assembly does not meet the quality requirements, and "1" indicates that the individual PCB assembly meets the quality requirements. Therefore, the inspection result indicates, for example, whether all connections between components are fully formed. The inspection result can also indicate whether the PCB assembly will work after its production. The inspection result of the AOI system is preferably calculated by means of a computer program, which is installed and / or executable on a computing unit such as a processor. This computer program may include image processing of one or more images of the PCBA and may output numerical measurement results. Based on the inspection result I1, the production of the PCBA can be controlled, for example, a conveyor belt on which the PCBA is placed can be controlled. The inspection result can also be displayed via a display unit D or a separate signaling device (for example, in the form of a signal light system). The inspection result I1 can indicate a fault or error-free state of the PCB assembly. In addition, the inspection result can represent multiple error categories. For example, the error category may indicate several different fault states (ie, errors) of the PCB assembly. In any case, the error category may also be used to indicate a false positive or pseudo error, as will be described later.
[0030] refer to Figure 2 , the benefits of AOI systems in terms of quality assurance and cost savings depend largely on the capabilities and quality of work of engineers and operators. This is due to: parameter tuning of different types of settings (e.g., camera / vision parameters, inspection algorithms, thresholds) is a truly complex challenge: the difference between images showing correct and faulty parts is usually only a few pixels, and finding the "correct" settings to address these tiny signals is not easy and may cause parts with defects to be "ignored". This is why engineers tend to be too strict, which also means that some parts will be rejected during the inspection process and need to be manually reprocessed at the quality station, even if these parts are defective. Therefore, quite frequently, up to 20%-30% of the inspected PCBAs are marked as "faulty" by the AOI system, of which only 10%-15% of these faulty marked objects are actual errors or true positives. Actual errors are errors that have been manually verified. Therefore, there is a constant trade-off between ensuring quality and reducing false positives. This is exacerbated by third-party factors such as external suppliers and differences in production steps before inspection, which may change and therefore force the AOI system to be updated.
[0031] Obviously, a high number of false positive classifications by the AOI system slows down the manufacturing process, reduces productivity (first-pass yield) and increases the overall production cost, mainly due to the additional manual effort of re-inspection.
[0032] False alarms (also called and labeled as pseudo errors or false positives) lead to non-conformity costs and increase production efforts because operators need to interrupt their current jobs (e.g., at the solder paste printer) to walk to the AOI system. In addition, manual handling of PCBAs and unnecessary visual inspections of products occur. Repeated manual inspections that do not result in successful identification of actual errors may cause operators to increase the rate of false detection. Entering the inspection results (i.e., true positives or true negatives) into the quality database also brings additional efforts. This leads to a decrease in overall equipment effectiveness (OEE) because manual inspections take time, can delay production, and lead to bottlenecks in the entire production (production line). Permanent bottlenecks at the AOI system 10 may require more testing capacity and may lead to unnecessary investments in technology upgrades or new test equipment. This leads to a reduction in the first pass rate (FPY), for example, by 15%, because each false alarm of the AOI system reduces the throughput. In addition, manual efforts are required by test engineers (especially for high-mix production) to optimize the test program of the AOI system.
[0033] Many different problems can occur with PCBAs. These include missing components, displaced or twisted wires, use of incorrect components, insufficient soldering, overly thick solder joints, bent IC pins, and lack of wetting. In order to eliminate these defects, careful inspection of assembled and soldered components is essential. Therefore, in order to improve the inspection, the AOI system can be upgraded with 3D AOI (instead of 2D) by using more modern cameras. But this is always accompanied by the creation and maintenance of inspection databases that control up to 100 parameters, and therefore the cost of ongoing expertise and complexity. Therefore, it is necessary to set up and (manually) maintain and fine-tune the AOI inspection algorithm, especially including the image processing of one or more images of the PCAB. This manual fine-tuning of the AOI system is both time-consuming and requires professional skills.
[0034] Reference now Figure 3, images of PCBA (captured by AOI system) and operator labels (obtained by manual inspection) can be used to train or retrain image-based machine learning models ML. The machine learning model can be used as a classifier that provides a binary label for a given test object (e.g., a region of interest in an image of a PCBA). The numerical measurement results of the AOI system can also be used instead of one or more images described in Thielen N., Werner D., Schmidt K., Seidel R., Reinhardt A., Franke J.: A Machining Learning Based Approach to Detect False Calls in SMT Manufacturing, 43rd International Spring Seminar on Electronics Technology, ISSE 2020 (Demanovska Valley, 14. Mai 2020-15. Mai 2020), In: Proceedings of the International Spring Seminar on Electronics Technology 2020, DOI: 10.1109 / ISSE49702.2020.9121044.
[0035] Therefore, in order to reduce the manual effort at the quality inspection and / or checkpoint, a machine learning model can be used, which filters the initially wrongly marked PCBAs. As a result, only these filtered cases are transferred to manual inspection, thereby reducing the effective workload of the operator. The remaining PCBAs are directly transferred to the next production step without any further delay. As a result, the (manual) testing effort is reduced, or equivalently, the overall first pass yield (FPY) is increased.
[0036] Now, refer to Figure 4In order to reduce the workload of manual inspection, it is envisaged that the false alarm reduction can also be extended, that is, it can be used for different inspection types. The inspection type can be given by a specific set of numerical inspection results (=AOI system output). Therein, different numerical inspection results can be obtained by different imaging techniques (when capturing one or more images of the PCBA) and / or by different image processing algorithms and / or by measuring different objects in the image of the PCBA (for example, measuring different components of the PCBA). For example, the numerical measurement results can be obtained by comparing the grayscale values of the pixels in the image(s). The inspection type may be accompanied by or may be equivalent to an error report generated and / or output by the AOI system. Since the inspection type may change depending on the configuration of the AOI system, it is necessary to provide a solution for reducing the false alarm rate covering multiple inspection types.
[0037] To this end, the software application may be logically divided into several microservices that run on an apparatus (such as an edge device) and provide inspection results and optionally provide a dashboard for displaying the inspection results.
[0038] Inspection of PCBAs can be performed before and / or after the soldering process. This is due to the use of numerical measurements generated by the AOI system instead of images captured by the AOI system. These numerical measurements are independent of the location of the AOI system within the production (production line).
[0039] In any case, the proposed solution (e.g. in the form of a software application) is seamlessly integrated into the process flow and is located on a device that can be placed near the AOI system. For example, the software application on the device receives as input the numerical measurement results generated by the AOI system (e.g. in the form of one or more files). Sub-services of the software application (such as data transmission, parsing and / or pre-processing) can be provided in order to achieve a compatible data format. Finally, a machine learning model in the form of a classifier can be executed for one or more or each PCBA under test. The software application including the classifier preferably provides a binary output, namely "real error" or "pseudo-error". This output can be used to control production, i.e., to start or prohibit an action in production, for example by selecting the correct track at a conveyor belt for the PCBA (under test). Here, the pseudo-error is directed to the next production step without further delay, while the real error is still forwarded to manual inspection. In addition, the software application can be used to display the real error on a user dashboard (i.e., as described above in conjunction with Figure 1 The inspection results are visualized and reported on a display as described.
[0040] To ensure the functionality and quality of the software application, the device's log data and / or numerical measurements and / or inspection results (from OAI and / or from classifiers) can be securely transmitted to cloud storage. This data can then be used for monitoring activities, for example, the functionality of all subcomponents of the software application (such as microservices), the performance of deployed machine learning models, data drift checks, and specific actions (e.g., operator notifications), and / or the need for retraining of machine learning models can be evaluated.
[0041] The training of the machine learning model as a classifier can be based on the available historical data set, i.e., the numerical measurements obtained from the AOI system and the error categories from the manual inspection stations. First, a data preprocessor can be trained, which can include one or more of the following steps: merging two data sets (one data set includes the numerical measurements and the other data set includes the corresponding misalignment categories, e.g., as labels for the numerical measurements), definition of unique identifiers, application of (customer-specific) filtering conditions, classifier-specific preparation and formatting of the numerical measurements, one hot encoding of categorical variables (i.e., error categories), performing data stratification feature building actions.
[0042] The purpose of this feature construction is to achieve a more comprehensive and generalized data set of information compared to image-based input. The machine learning model will then use this data set to provide more accurate classification decisions and thus inspection results. The numerical measurement results of the AOI system are generated by analyzing a given one or more images of the PCBA and targeting specific details of the PCBA components. The image processing determines typical modes of failure, such as due to assembly and / or welding, and converts them into numerical measurements. Therefore, these numerical measurements carry specific expertise and are therefore used to further determine the quality of the PCBA. It should be understood that different inspection types desired may be used, depending on the components used and / or the level of detail of the inspection, and the AOI system may be configured accordingly to perform the different inspection types.
[0043] Furthermore, the numerical measurement results can be enriched or augmented with meta-information or metadata in order to build a feature set. This additional information makes the classification results better. The metadata can include one or more of the following: (classified) AOI system error description or error category ("Tin missing", "No meniscus", "Reverse polarity"...), PCB type, part number, inspection type, component type (due to the large number of unique types, it is preferably preferred not to use the original format, but to classify into functionally similar groups, like "resistors", "capacitors", etc.), measurement type ("body" / "component" or "pin" position).
[0044] Figure 4An illustration of an AOI system and an apparatus communicatively connected to the AOI system, wherein the apparatus is used to verify inspection results of the AOI system. Therein, a software application evaluates PCBAs (more precisely, corresponding numerical measurements) that have been determined to be faulty by the AOI system. Thereby, the software application reduces the number of PCBAs to be inspected by an operator at a repair station. The operator can then manually inspect the reduced set of PCBAs that the software application has verified as failed, and can mark the PCBAs as false positives or actual errors, and these results can be stored in a computer as described above in conjunction with Figure 1 The quality described in the database.
[0045] Reference now Figure 5 , in order to achieve generalization (i.e. make the classification independent of a specific inspection type), the following is proposed. For different AOI system inspection types (accompanied by a specific set of numerical inspection results (=AOI system output)), the following reshaping process is proposed to create a unified comprehensive "inspection space" or feature vector: The feature vector is assigned by the concatenation of numerical measurements (of one or more inspection types). Therein, non-existent inspection features or numerical measurements are left blank or as NaN. Then, the non-existent inspection features are filled with the average value. To this end, a normalization step is implemented by taking statistical measures like "minimum, maximum, average, quantile" and "value count" for each numerical measurement result and metadata category, respectively, for different component types (possibly with varying numbers of pins).
[0046] Finally, the dimension of the feature vector (similar to a huge sparse matrix) is reduced, for example, via principal component analysis (PCA). Typically, volume reduction techniques project a set of related variables into a smaller set of unrelated variables. Specifically, PCA achieves the above by finding the direction of the maximum variance in the original high-dimensional data and projecting them into a smaller dimensional space. The maximum variance is given for the most frequent inspection types because they are the most measured and filled into the corresponding columns of the feature vector. As a result, the trained dimensionality reduction model can be applied as long as the frequency of the inspection type does not change significantly. Nevertheless, this situation is very unlikely because, first of all, the training is carried out on a huge data set with a saturated frequency number. Secondly, preprocessing and / or retraining of the classifier may occur very frequently, for example, once a new label arrives. Therefore, retraining will take into account the changed inspection type frequency. This generalization process allows standardization of machine learning features in order to cope with changes from the output of the AOI system.
[0047] The output of preprocessing is a “model-ready” data format (i.e., input features) that will be consumed by a machine learning model (i.e., a classifier).
[0048] To train the classifier, the following optimization functions are provided:
[0049] min(k*slip_rate)+max(test_effort_reduction).
[0050] The optimization function adjusts the weights and / or biases of a machine learning model used as a classifier.
[0051] Among them, two metrics are calculated directly from the confusion matrix cf. Figure 7 is the following binary classifier:
[0052] False positive rate = false positive / (false positive + true negative)
[0053] Test workload reduction = true positives / (true positives + false negatives)
[0054] The optimization task operates on a set of parameters, mainly model parameters and class decision thresholds. It is also proposed to select a class-imbalanced perceptual classifier, since the number of real errors is typically 10-20 times less than pseudo errors. For example, a cost-sensitive classifier can be used or sampling techniques can be applied to the data. The optimal parameter values are adopted to create a trained machine learning model, i.e., a classifier.
[0055] refer to Figure 6 , shows a (binary) classifier. A classifier is a machine learning model that assigns output categories or groups to input features. In the current case, the classifier is a binary classifier that divides the input into two categories, namely, true errors or pseudo errors. The output of the classifier can be the probability of pseudo errors. Figure 6 In FIG. 1 , a neural network is shown, and other machine learning models, such as random forests, may also be used as classifiers. Input features of the classifier are obtained from the AOI system. Specifically, the input features of the classifier are obtained by reducing the dimension of a feature vector (e.g., using PCA). As explained herein, a feature vector is a data structure including a plurality of parts, wherein each part corresponds to an inspection type of the AOI system, preferably all inspection types of the AOI system.
[0056] exist Figure 7 In the figure, the confusion matrix of the classifier is shown, based on which the number of true negative TN, false negative FN, false positive FP and true positive TP can be determined. Based on these results, the metric of the optimization function cf of the classifier can be calculated, i.e., min(k*slip_rate)+max(test_effort_reduction) above, and the training and optimization of the classifier can be performed.
[0057] refer to Figure 8, describes the operation phase, where the software application is deployed on an apparatus (e.g. an edge device). Once the data exporter receives a new file comprising numerical measurements and / or metadata, the trained data preprocessor converts the numerical measurements and / or metadata into a model-ready format. Finally, the classifier is applied and an error class can be output and can be used to control, e.g., a conveyor belt accordingly and / or also visualized on a display and / or (operator) dashboard.
[0058] Numerical measurements are generated by the internal algorithms of the AOI system from one or more images of the PCBA. This conversion depends on a large number of parameters as described above. Therefore, the calculated numerical measurements can be more properly understood as "features" that have applied several logical steps of image interpretation, through which typical errors of the PCBA are checked. Depending on the AOI system supplier, some of these features already appear as a string of granular (sub) inspection tests, and all together create a single integrated decision, i.e., the inspection result of the AOI. These features characterize certain test object states, and therefore, fault states are also accompanied by specific patterns within this feature space. Therefore, using these numerical measurements as features and building logic on their basis provides benefits.
[0059] The proposed solution achieves better scalability across different AOI system inspection types and AOI system suppliers. As mentioned earlier, from a model input perspective, most AOI systems come with certain feature sets (and although generated by different logic), the feature structure is highly similar, making it possible to introduce a common data standard that covers all inspection types and / or suppliers. The proposed solution is built in this standardized format (i.e., feature vector) and thereby broadens the path to scalability. On the other hand, depending on the inspection type and / or supplier, the image can differ in format and shape. This is not only valid for different suppliers, but also for different component types, both of which are inspected by the same AOI system.
[0060] Therefore, when faced with different data input formats due to a given and specific AOI system vendor or manufacturer, the measurement space is not fixed, but varies depending on the manufacturer-specific "inspection type". This is overcome by the introduction of feature vectors, which "normalize" all volatile numerical measurement results and are therefore valid for all inspection types and / or component types. In addition, most AOI system manufacturers allow the generation of additional measurements, and thus inspection types, beyond what is provided ex-factory. This extension is crucial in order to allow machine learning methods to be more precise and meet the needs of PCBA inspection.
[0061] The proposed solution caters to individual PCBA production needs and is not just a "standardized" machine learning model, it also tries to optimize a general accuracy metric (not using customized cases and not recommended in real world problems where class imbalance often exists by default). This is achieved by not only doing a "random search" of hyperparameter tuning, but by performing a hyperparameter search based on use case specific optimization as will be described below.
[0062] First, choose a loss function: For example, instead of using the standard (but still ignoring class imbalance) cross entropy loss to train a machine learning model (such as a classifier), you can use a focal loss (Focal Loss) that is aware of class imbalance (as described by TSUNG-YI LIN et al. in Focal Loss for Dense Object Detection, https: / / doi.org / 10.48550 / arXiv.1708.02002). This loss is a generalization of the cross entropy loss, which allows to focus more specifically on the minority class by adjusting two additional parameters alpha and gamma and by ensuring more robust class imbalance handling.
[0063] Secondly, customizable hyperparameters for the optimization function are introduced: As shown above, in addition to the two metrics "slip" and "test_effort_reduction", the penalty factor k (i.e., a customizable hyperparameter) specifies the emphasis on minimizing the false positive rate. By optimizing this optimization function, we ensure that the optimal machine learning model, i.e., classifier (parameterization) is selected. It should be understood that the optimal machine learning model or classifier has individual characteristics and depends on individual preferences regarding the acceptance threshold of the false positive rate. In fact, this threshold may even vary for different types of components and related requirements such as safety, for example, when producing safety-critical or less critical PCBAs.
[0064] AOI systems can use ultraviolet light, X-rays, and infrared light for inspection. Therefore, it is possible to determine defects by the measured component height profile (such as elevated component body / body coplanarity) or by the measured pin shape profile (such as elevated pin / pin coplanarity), or by the measured height and volume information of all pixels in the connection area (such as solder joint defects). AOI systems can therefore measure the Z dimension of PCBAs including electronic components and solder joints. AOI systems can also provide markings and quantify defects such as missing pins, offsets, rotations, polarity, inversions, OCVs, welds, kanban effects, elevated pins, elevated bodies, tombstone effects, bridges, etc.
[0065] The imaging system of an AOI system may include a single camera, or there may be more than one camera to provide the possibility of better imaging and 3D capabilities. The cameras should also be able to move under software control. This will enable them to be moved to the optimal position for a given PCB assembly.
[0066] However, the problem of false positives persists, for example due to the flexible environment of PCBA production. In addition, the PCB itself may warp, which occurs especially after the soldering process, which leads to changed component positions and thus false positives.
[0067] After the acquisition of the numerical measurement results is completed, the data is exported from the AOI system (for example) into an ASCII readable format. For example, the numerical measurement results may include the part name and an indication of whether the inspection result is a pass or fail, i.e., the error category (for example in the form of metadata).
[0068] In any case, the AOI system may operate according to a test program. The test program may include one or more inspection types to be performed. In order to perform one or more inspections (of one or more inspection types), different measurements need to be implemented. The test program and therefore these inspection types may include an indication of one or more regions of interest on the PCBA. Further information about the inspection type may include the PCB type, one or more component types within the region of interest. The test program of the AOI system may then output numerical measurement results (including quantification of defects for each inspected PCBA) and error categories cf. Thielen N., Werner D., Schmidt K., Seidel R., Reinhardt A., Franke J.: AMachine Learning Based Approach to Detect False Calls in SMT Manufacturing, 43rd International Spring Seminar on Electronics Technology, ISSE 2020 (De-manovska Valley, 14. Mai 2020-15. Mai 2020), In: Proceedings of the International Spring Table 1 of Seminar on Electronics Technology 2020, DOI: 10.1109 / ISSE49702.2020.9121044.
[0069] As described above, the classifier may be a binary classifier, such as a preferably balanced random forest. The classifier may be trained and / or used during the production of the PCBA. The classifier may provide as output an indicator indicating a "real error" or a "pseudo-error". The output of the classifier may be used as an inspection result that controls the conveyor (belt). In the case of a real error, a manual inspection is initiated, while in the case of a pseudo-error, the next production step may be initiated.
[0070] The output of the classifier may be related to only one region of interest, such as an object in an image of a PCBA (corresponding to a component of the PCBA). However, the inspection result of the classifier may be assigned to the entire PCBA. The classified information may be aggregated in different ways relative to the hierarchy of the PCBA's data.
[0071] As described above, a classifier, such as in the form of a random forest, is optimized relative to an optimization function, as described above. The training of the classifier can be done in the cloud, and the trained classifier and the trained preprocessing can then be deployed on a device such as an edge device.
[0072] For example, once the API of the AOI system is ready or the XML file on the AOI system is ready, the verification of the inspection results of the AOI system can begin. In order not to disturb the production setup of the PCBA, performing the verification preferably takes less time than the time between the AOI system inspecting two consecutive PCBAs.
[0073] As described above, after verification, further testing (such as in-circuit testing) may follow, or the PCBA may be installed in a housing of a device to be produced.
[0074] refer to Fig. 9 , showing exemplary steps according to a first embodiment. In step S1, a numerical measurement result is obtained. The numerical measurement result belongs to one or more inspection types of a plurality of inspection types of an automated optical inspection AOI system. The numerical measurement result is generated by the AOI system, for example, as a result of image processing of one or more images of the PCBA. The image may be captured or acquired by an imaging system of the AOI system.
[0075] In step S2, the numerical measurement results are entered into at least one part of the feature vector. Wherein, each part of the feature vector can be associated with a corresponding inspection type. The AOI system may include multiple inspection types, each inspection type including one or more measurements that produce numerical measurement results. Different inspection types vary based on different measurements performed by the AOI system. Each inspection type can be identified by a (unique) identification number, based on which the inspection type can be assigned to a corresponding part in the feature vector. Therefore, the feature vector includes multiple parts, each of which is associated with a corresponding inspection type in multiple inspection types. The feature vector can be filled with numerical measurement results that have been performed by the AOI system operating according to the test program. The test program can be configured and can determine the inspection type and therefore the measurement performed by the AOI system. Of course, not all inspection types and therefore the measurements are performed by the AOI system in the configuration settings - but only those configured according to the test program. Unperformed inspection types and therefore the parts of the feature vector that are not entered or filled with numerical measurement results are left blank, or dummy data such as NaN are inserted into those parts. For example, after dimensionality reduction of the feature vector, missing input features may be filled with the mean value of the missing numerical measurements.During the training phase, the mean value of the numerical measurements may be or may have been determined based on a training data set, which includes the necessary numerical measurements.
[0076] In step S3, input features from the feature vector are selected by dimensionality reduction of the feature vector. As described herein, PCA can be used to reduce the dimension of the feature vector. Principal component analysis PCA performs a linear mapping of the feature vector to the input features. This can be understood as a selection. Of course, other dimensionality reduction methods can also be used, including, for example, nonlinear kernels.
[0077] In step S4, the input features may be input into a classifier. The previously trained classifier may be capable of determining the error category of the PCBA based on the input features. The error category may be a pseudo-error or a real error and may be determined based on the output of the classifier. As described above, the output may be a probability for the classification and assignment of the error category.
[0078] In step S5, the error category can be output as the inspection result of the PCBA. Therefore, the inspection result of the AOI system can be replaced by the inspection result of the classifier. The inspection result can therefore be used to control the PCBA production and / or can be displayed on a display. Then, the operator can manually inspect the tested PCBA.
[0079] refer to Fig.10, showing an explanation of exemplary steps according to the second embodiment. In step S0, one or more images of the PCBA are processed using image processing. Therefore, based on the image processing of one or more images of the PCBA, a numerical measurement result is obtained. The image processing may be based on a grayscale comparison of image pixels, for example, compared with a golden image. Thus, a geometric characteristic (such as length, area, volume or angle, and / or brightness or color) of at least one object in one or more images of the PCBA may be obtained as a numerical measurement result. As mentioned, these features may indicate a defect of the PCBA. Step S0 may be performed before step S1. Again, as previously described, in step S2, the obtained numerical measurement result is entered into at least one part of the feature vector. The numerical measurement result corresponds to a measurement variable of the inspection type, and / or wherein different inspection types include different measurement variables. Then, in step S6 (which may be part of step S3 for selecting input variables from the feature vector), the feature vector is mapped to the input feature. Wherein, the mapping may be based on a principal component analysis of the feature vector. Then, it may be step S4 as described above.
[0080] Fig.11 An illustration of exemplary steps according to a third embodiment is shown. In step S7, reference values of numerical measurement results and / or associated error categories are obtained. These reference values (and / or associated error categories) can be used to train a classifier. Therefore, in step S8, a classifier can be trained (based on) the reference values of numerical measurement results associated with the error categories. As described above, training can include a loss function and an optimization function. Therefore, a trained classifier is obtained.
[0081] Fig.12 An illustration of exemplary steps according to a fourth embodiment is shown. In step S9, the numerical measurement results are augmented with metadata. For example, they may be stored together or may be associated in other ways. The metadata may relate to one or more components of the PCBA, one or more inspection types, and / or inspection results of the one or more inspection types.
[0082] In step S10, the metadata may be input into a classifier. The classifier is capable of determining the error category of the PCBA based on the input features and the metadata. To this end, the classifier may have been previously trained based on reference values of numerical measurement results and metadata associated with the error category.
[0083] Therefore, in step S11, the error type may be output as the inspection result.
[0084] Fig.13An illustration of exemplary steps according to a fifth embodiment is shown. In step S12, numerical measurements are obtained, input features are selected and / or input into a classifier (only if) the AOI system has determined an error with the PCBA.
[0085] In step S13, further processing of the PCBA can be controlled based on the inspection results according to the error category determined by the classifier, for example, by controlling a conveyor on which the PCBA is placed. The error category determined by the classifier represents a false error or a real error of the inspection of the AOI system.
[0086] Fig.14 An illustration of a device 25 including an interface 20, a processor 22 and a memory 21 is shown. The interface 20 may be wired or wireless and may be used to exchange data between the AOI system and the device 25. The interface 20 may be connected to Figure 1 In any case, the numerical measurement results can be obtained through the interface. For example, the AXI system can generate a file including the numerical measurement results. The file can then be obtained through the device 25. The device 25 may include a software application executed by the processor 22 and the memory 21 of the device. The software application may be based on microservices, where each microservice performs a dedicated function. For example, the microservice may execute as described above in particular with respect to Figures 9 to 13 The software application may now perform the functions of a data exporter, a data preprocessor. The software application may also include a classifier. The software application may also output or initiate or trigger control commands for controlling the conveyor (belt) and / or may display the inspection results of the classifier on a display. Thus, the software application is computer-implemented and operable to perform the steps described herein, in particular, the steps related to Figures 9 to 13 , and preferably uses the memory and processor of the device. The device 25 may include a housing in which the memory 21 and the processor 22 are located. The device 25 is preferably external to the AOI system, but may also be integrated into the AOI system.
[0087] Fig.15 An illustration of exemplary steps according to a sixth embodiment is shown. In step S14, reference values of numerical measurement results are obtained from the AOI system. The reference values may be stored, for example, in a cloud computing environment, in which the setting of the classifier and the data preprocessor is also performed.
[0088] In step S15, the reference values of the numerical measurement results are marked with error categories. The error categories may be obtained by and / or after manually inspecting one or more PCBAs.
[0089] In step S16, the classifier is trained or retrained based on the reference values and the error categories by optimizing an optimization function. The optimization function comprises: a false detection rate, which is given by the ratio between the following test results: false positives on the one hand, and true negatives and false positives on the other hand; and / or a test reduction, which is given by the ratio between the following test results: true positives on the one hand, and true positives and false negatives on the other hand. As specifically described above in conjunction with Figure 7 As explained, when examining the output of the classifier, the number of true negatives, true positives, false negatives and false positives can be obtained through the confusion matrix. As mentioned above, the optimization function may also include a penalty factor k.
Claims
1. A computer-implemented method for inspecting a printed circuit board assembly (1) PCBA, comprising the following steps: Obtaining (S1) numerical measurement results (Feat 1, ... Feat n) of one or more inspection types from a plurality of inspection types of an automated optical inspection (AOI) system (10), wherein the numerical measurement results are generated by the AOI system, entering (S2) the numerical measurement result into at least one part (a1, a2, an) of a feature vector (FV) associated with the one or more examination types, the feature vector (FV) comprising a plurality of parts, each of the plurality of parts being associated with a respective examination type of the plurality of examination types, selecting (S3) input features (IF) from said feature vector (FV) by reducing the dimension of said feature vector (FV), inputting (S4) the input features (IF) into a classifier, wherein the classifier is capable of determining an error category (EC) of the PCBA (1) based on the input features (IF), The error category (EC) is output (S5) as the inspection result of the PCBA (1).
2. The method according to the preceding claim, wherein: The step of selecting the input features (IF) comprises mapping (S6) the feature vector (FV) onto the input features (IF), wherein, preferably, the mapping is based on a principal component analysis of the feature vector (FV).
3. A method according to any one of the preceding claims, wherein: The numerical measurement result is obtained based on image processing (S0) of one or more images of the PCBA (1).
4. A method according to any one of the preceding claims, wherein: The classifier (60) is previously trained (S8) based on reference values of the numerical measurement results associated with the error class (EC).
5. The method according to the preceding claim further comprises the following steps: augmenting (S9) the numerical measurement result with metadata related to one or more components of the PCBA (1), one or more inspection types and / or inspection results of the one or more inspection types, inputting (S10) the metadata into the classifier (60), wherein the classifier is capable of determining the error category of the PCBA based on the input features and the metadata, and wherein the classifier is pre-trained based on reference values of the numerical measurement results and metadata associated with the error category, and The error category (EC) is output (S11) as a check result.
6. The method according to any one of the preceding claims, further comprising the steps of: obtaining numerical measurements related to geometrical properties, such as length, area, volume or angle, and / or brightness or color, of at least one object in one or more images of the PCBA (1), and / or Therein, the numerical measurement result corresponds to a measurement variable of an examination type, and wherein different examination types include different measurement variables.
7. The method according to any one of the preceding claims, further comprising the steps of: only if the AOI system has determined an error in the PCBA, obtaining (S12) the numerical measurement result, selecting the input feature (IF) and / or inputting the input feature into the classifier, and / or wherein the error category determined by the classifier indicates a false error or a real error of the inspection performed by the AOI system, and / or Based on the inspection result, further processing of the PCBA is controlled ( S13 ), for example, by controlling a conveying device on which the PCBA is placed.
8. The method according to any one of the preceding claims, further comprising the steps of: After the image processing of the one or more images of the PCBA is completed, for example according to one or more inspection types of the AOI system (10), the numerical measurement results are obtained from the AOI system (10).
9. A method according to any one of the preceding claims, wherein: The processing time of the numerical measurement results until the classifier outputs the inspection results is less than a cycle time for inspecting two subsequent PCBAs by the AOI system.
10. An apparatus (25) capable of performing the steps of the method of any of the preceding claims.
11. An apparatus (25) comprising a first interface (20) for obtaining numerical measurement results of one or more inspection types of a plurality of inspection types of the automated optical inspection (AOI) system from an automated optical inspection (AOI) system (10), wherein: The numerical measurement result is generated by the AOI system, The apparatus further comprises a processor (22) and a memory (21) for entering the numerical measurement result into at least one part of a feature vector associated with the one or more examination types, the feature vector comprising a plurality of parts, each of the plurality of parts being associated with a respective examination type of the plurality of examination types, for selecting input features from the feature vector by reducing the dimension of the feature vector, for inputting the input features into a classifier, wherein the classifier is capable of determining an error category of the PCBA based on the input features, and Used to output the error category as the inspection result of the PCBA.
12. A method of using the method according to claim 1 and / or the device according to claim 11 for subsequent inspection of a PCBA (1) and / or for determining false errors of an AOI system (10).
13. A method of training or retraining a classifier (60) for PCBA inspection, preferably during production of a plurality of PCBAs, obtaining (S14) a reference value of a numerical measurement result from the AOI system; For example, after manually checking the PCBA (1), marking (S15) the reference measurement result with an error category, and The classifier (60) is trained or retrained (S16) based on the reference value and the error class (EC) by optimizing an optimization function, the optimization function comprising: The false positive rate, given by the ratio between false positives (FP) on the one hand and true negatives (TN) and false positives (FP) on the other hand, and / or Test reduction, given by the ratio between test results: true positives (TP) on the one hand, and true positives (TP) and false negatives (FN) on the other hand.
14. A method of using (S17) a classifier (60) obtained by the method according to claim 13 for subsequent inspection of PCBA and / or for determining false errors (FP) of an AOI system (10).
Citation Information
Patent Citations
Method and apparatus for rapid classification of sub-components of manufactured component
CN114207624A
Cited By
Determination method of first pass yield and anomaly analysis method based on first pass yield
CN121186489A
Device and method for using deep learning model to re-inspect defective solder paste printed circuit boards
TWI903894B
