An automated system for packaging substrate detection

Abstract

The present invention discloses an automated system for packaging substrate inspection, which relates to the field of data encryption technology. The system comprises a data acquisition module for collecting and preprocessing multi-view image data; a feature extraction module for extracting features from the preprocessed multi-view image data; a feature map fusion module for aligning and fusing the extracted feature maps using an image registration algorithm to form a three-dimensional feature map of the substrate; a defect probability prediction module for constructing a defect recognition model and predicting the substrate defect probability based on the substrate's three-dimensional feature map; and a defect grade assessment module for predicting the substrate defect degree score based on the substrate defect probability, assessing the substrate defect grade based on the substrate defect degree score, and generating a substrate inspection report. The present invention implements intelligent identification and assessment of multiple defect types, significantly improving the accuracy and automation level of inspection.

Description

Technical Field

[0001] The present invention relates to the technical field of package substrate detection, in particular to an automated system for package substrate detection. Background Art

[0002] With the increasing integration of electronic products, the quality and reliability of packaging substrates directly impact product performance and lifespan. Therefore, efficient and accurate inspection of packaging substrates has become a key research topic in the electronics manufacturing field. Traditional packaging substrate inspection methods primarily rely on artificial vision or optical inspection equipment, which often suffer from slow inspection speeds, insufficient accuracy, and difficulty identifying complex defects. In recent years, with the advancement of machine vision and deep learning technologies, automated substrate inspection systems have gradually become a research hotspot. By combining multi-view image acquisition, feature extraction, and intelligent algorithms, automated inspection systems can complete comprehensive inspections of substrate surfaces and internal structures in a short period of time. They also have the ability to identify and evaluate a variety of complex defects (such as poor solder joints and packaging layer damage), significantly improving inspection efficiency and accuracy.

[0003] However, existing automated substrate inspection systems still have some shortcomings in practical applications. First, most existing systems rely solely on two-dimensional images for inspection and lack in-depth analysis of the substrate's three-dimensional structure, making it difficult to accurately identify certain complex spatial defects (such as hidden defects in multi-layer structures). Second, existing defect recognition methods often only address a single defect type and lack the comprehensive ability to uniformly evaluate multiple defect types. In particular, the accuracy and reliability of existing methods need to be improved in the quantitative assessment of defect severity. Therefore, how to construct a three-dimensional feature map of the substrate through multi-view image fusion technology and accurately identify and assess the severity of multiple defects based on this three-dimensional feature map has become a pressing issue in the field of automated inspection. Summary of the Invention

[0004] In view of the above existing problems, the present invention is proposed.

[0005] Therefore, the present invention provides an automated system for packaging substrate inspection to solve the problems in the prior art that two-dimensional inspection cannot accurately identify complex spatial defects and the defect severity assessment is inaccurate.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0007] In a first aspect, the present invention provides an automated system for packaging substrate inspection, which includes a data acquisition module for acquiring multi-view image data and preprocessing the multi-view image data; a feature extraction module for extracting features from the preprocessed multi-view image data; a feature map fusion module for aligning and fusing the feature maps based on the extracted feature maps through an image alignment algorithm to form a three-dimensional feature map of the substrate; a defect probability prediction module for constructing a defect recognition model to predict the substrate defect probability based on the three-dimensional feature map of the substrate; a defect level assessment module for predicting the substrate defect degree score based on the substrate defect probability, assessing the substrate defect level based on the substrate defect degree score, and generating a substrate inspection report.

[0008] As a preferred solution of the automated system for package substrate detection of the present invention, wherein: the multi-view image data includes image data of the solder joints, edges, package layers and surfaces of the substrate

[0009] As a preferred solution of the automated system for packaging substrate inspection of the present invention, the multi-view image data is pre-processed in the following specific steps:

[0010] The collected multi-view image data is subjected to noise filtering through Gaussian filters;

[0011] Enhance image contrast through adaptive histogram equalization;

[0012] Use SIFT to spatially align images from different perspectives;

[0013] The effective area of ​​the substrate is separated from the background through the edge detection algorithm.

[0014] As a preferred solution of the automated system for packaging substrate inspection of the present invention, wherein: the feature extraction of the pre-processed multi-view image data is performed in the following specific steps:

[0015] The circular Hough transform algorithm is used to identify the solder joint area in the image and extract the geometric shape feature map and texture feature map of the solder joint;

[0016] The contour feature map of the substrate edge is extracted using the Canny edge detection algorithm;

[0017] GLCM is used to extract the texture feature map of the substrate packaging layer area.

[0018] As a preferred solution of the automated system for packaging substrate detection described in the present invention, wherein: based on the extracted feature map, the feature map is registered and fused by an image registration algorithm to form a three-dimensional feature map of the substrate. The specific steps are as follows:

[0019] Based on the extracted geometric shape and texture feature maps of the solder joints, the contour feature maps of the substrate edge, and the texture feature maps of the package area, SIFT is used to match feature points at the same position in the feature maps from different viewpoints.

[0020] Use affine transformation to calculate the geometric transformation matrix between adjacent perspectives and accurately align feature maps from different perspectives;

[0021] A weighted average fusion algorithm is used to fuse the feature information of the same feature map at different viewing angles, and an occlusion detection algorithm is used to identify blind spots at each viewing angle to eliminate redundant parts between different viewing angles.

[0022] Based on the fused feature map, a structured light reconstruction algorithm is used to generate three-dimensional point cloud data;

[0023] Based on the three-dimensional point cloud data, a surface reconstruction algorithm is used to convert the point cloud data into a three-dimensional feature map of the substrate.

[0024] As a preferred solution of the automated system for packaging substrate detection described in the present invention, wherein: the defect recognition model is constructed to predict the probability of substrate defects based on the three-dimensional feature map of the substrate, and the specific steps are as follows:

[0025] 3D-CNN is selected as the basic model. The 3D convolution kernel is used to extract local feature information from the 3D feature map of the substrate. The pooling layer is used to reduce the dimension of the 3D feature map of the substrate and retain important features. The fully connected layer is used for final classification to form the initial defect recognition model.

[0026] Based on historical 3D substrate data, the initial defect recognition model is trained using a stochastic gradient descent optimization algorithm. The network parameters are adjusted to minimize the loss function to generate the final defect recognition model.

[0027] The three-dimensional feature map of the substrate is input into the defect recognition model to perform defect recognition on the substrate. The expression is:

[0028]

[0029] in, is the three-dimensional feature map of the substrate at the i-th viewing angle, is a nonlinear activation function, Represents the output features after being mapped by the feature mapping function at the i-th perspective, is a three-dimensional convolutional neural network, is the classification function, is the number of three-dimensional feature maps of the substrate, i is the feature index of the three-dimensional feature map of the substrate, represents the defect probability of the jth defect category, where j is the index variable of the defect type.

[0030] As a preferred solution of the automated system for packaging substrate detection of the present invention, wherein: the substrate defect degree score is predicted based on the substrate defect probability, and the specific steps are as follows:

[0031] According to the probability of each defect Calculate the corresponding severity score. The severity of each defect category is nonlinearly mapped using the hyperbolic tangent function. The Gaussian kernel function is used to dynamically adjust the impact of each defect type on the total severity score of the substrate defect to obtain the final substrate defect severity score. The expression is:

[0032]

[0033] Where S is the severity score of the substrate, is the total number of defect categories, is the typical probability of defect type j, is the standard deviation of the probability of defect of type j, is the mapping slope control parameter of the j-th defect, is the mapping offset control parameter for the j-th type of defect.

[0034] As a preferred solution of the automated system for packaging substrate inspection of the present invention, wherein: the substrate defect level is evaluated according to the substrate defect degree score, and the specific steps are as follows:

[0035] Define the mild defect threshold s1 and severe risk threshold s2 based on historical data;

[0036] when When ≤s1, ​​the substrate defect level is considered to be a mild defect;

[0037] When s1< When <s2, the substrate defect level is considered to be moderate;

[0038] when When ≥s2, the substrate defect level is considered to be severe.

[0039] As a preferred solution of the automated system for packaging substrate inspection described in the present invention, the substrate inspection report includes defect detection results, severity scores, defect level classification, three-dimensional feature maps of the substrate, defect distribution statistics, substrate defect levels, and defect repair suggestions.

[0040] As a preferred solution of the automated system for packaging substrate detection described in the present invention, when a linear transformation is used to perform feature mapping on the input three-dimensional feature map of the substrate at the i-th viewing angle, the feature mapping function The expression is:

[0041]

[0042] in, is the output feature after feature mapping at the i-th perspective, is the weight matrix of the i-th perspective, is the bias term of the i-th perspective;

[0043] The expression of the classification function f is:

[0044]

[0045] in, is the raw score output by the 3D-CNN output layer for the j-th defect type, is the probability of the j-th type of defect calculated by the Softmax function, It represents the original score output by the 3D-CNN output layer for the k-th defect type, where k is an index variable of the defect type used only in the exponential summation process.

[0046] The beneficial effects of the present invention are as follows: through the feature map fusion module, feature map registration and fusion based on multi-view images are realized, and a three-dimensional feature map of the substrate is generated, so that the spatial structure of the substrate can be captured, and complex spatial defects can be accurately identified, overcoming the limitations of traditional two-dimensional detection; in addition, the defect probability prediction module constructs a defect recognition model based on a three-dimensional convolutional neural network (3D-CNN), and makes probability predictions for different defect types, solving the problem of single defect type identification in the existing technology, thereby realizing intelligent identification and evaluation of multiple defect types, and significantly improving the accuracy and automation level of detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0048] Figure 1 Schematic diagram of the automated system for packaging substrate inspection in Example 1.

[0049] Figure 2 This is a schematic diagram of preprocessing multi-view image data in Example 1. DETAILED DESCRIPTION

[0050] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0051] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0052] Secondly, the term "one embodiment" or "embodiment" herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in various places throughout this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive of other embodiments.

[0053] Example 1, reference Figure 1 and Figure 2 , which is a first embodiment of the present invention, provides an automated system for packaging substrate inspection, comprising:

[0054] The data acquisition module is used to acquire multi-view image data and pre-process the multi-view image data.

[0055] The multi-view image data includes image data of solder joints, edges, packaging layers, and surfaces of the substrate;

[0056] Furthermore, the above data is collected by industrial cameras or 3D sensors installed at different angles;

[0057] The collected multi-view image data is subjected to noise filtering through Gaussian filters;

[0058] For example, a Gaussian filter uses a weighted average to smooth an image, removing high-frequency noise while retaining key edge information. A Gaussian filter uses a weighted average to smooth an image, removing high-frequency noise while retaining key edge information.

[0059] Enhance image contrast through adaptive histogram equalization;

[0060] For example, adaptive histogram equalization adjusts the image brightness distribution based on local areas, enhancing contrast and detail visibility.

[0061] Use SIFT to spatially align images from different perspectives;

[0062] For example, the SIFT algorithm is used to detect and match key points in an image, calculate the geometric transformation matrix, and achieve precise alignment of multi-view images.

[0063] The effective area of ​​the substrate is separated from the background through the edge detection algorithm.

[0064] For example, significant edges in the image are determined by edge detection algorithms, thereby isolating the effective area of ​​the substrate and eliminating background interference.

[0065] The feature extraction module is used to extract features from the preprocessed multi-view image data.

[0066] The circular Hough transform algorithm is used to identify the solder joint area in the image and extract the geometric shape feature map and texture feature map of the solder joint;

[0067] Furthermore, after using the circular Hough transform algorithm to identify the solder joint area, the geometric shape features (such as diameter and roundness) and texture features (such as surface roughness and texture direction) of the solder joint are extracted. By combining geometric and texture information, more accurate and rich feature support is provided for the comprehensive evaluation of solder joint quality and defect detection.

[0068] The contour feature map of the substrate edge is extracted using the Canny edge detection algorithm;

[0069] Furthermore, through the Canny edge detection algorithm, the image is first denoised by Gaussian filtering, then the image gradient is calculated and the edges are refined using the non-maximum suppression method, followed by screening strong and weak edges through double thresholds, and finally a complete contour feature map of the substrate is generated through edge connection, providing accurate edge information for subsequent detection.

[0070] GLCM is used to extract the texture feature map of the substrate packaging layer area.

[0071] Furthermore, the texture feature map of the substrate packaging layer area is extracted through the gray-level co-occurrence matrix (GLCM). First, the packaging layer image is grayscaled, and then the co-occurrence frequency of pixel pairs with different grayscale values ​​is calculated. Then, texture features such as contrast, energy, and correlation are extracted from it to help describe the roughness and consistency of the packaging layer, providing important texture information for subsequent defect detection.

[0072] The feature map fusion module is used to align and fuse the feature maps based on the extracted feature maps through an image registration algorithm to form a three-dimensional feature map of the substrate.

[0073] Based on the extracted geometric shape and texture feature maps of the solder joints, the contour feature maps of the substrate edge, and the texture feature maps of the package area, SIFT is used to match feature points at the same position in the feature maps from different viewpoints.

[0074] Furthermore, key points are detected in feature maps of different perspectives, and a feature descriptor that is not affected by scaling, rotation, and illumination changes is generated for each key point; then, the Euclidean distance between these descriptors is calculated to find matching feature points at the same position in images of different perspectives; finally, the optimal matching point is screened out through methods such as nearest neighbor matching or ratio testing.

[0075] Use affine transformation to calculate the geometric transformation matrix between adjacent perspectives and accurately align feature maps from different perspectives;

[0076] Furthermore, through affine transformation, the matched feature point pairs are used to calculate the geometric transformation matrix (including rotation, translation and scaling) between adjacent perspectives, and the matrix is ​​applied to accurately align the feature maps of different perspectives to the same coordinate system.

[0077] A weighted average fusion algorithm is used to fuse the feature information of the same feature map at different viewing angles, and an occlusion detection algorithm is used to identify blind spots at each viewing angle to eliminate redundant parts between different viewing angles.

[0078] It should be noted that the weighted average fusion algorithm assigns different weights to the feature maps of each view. These weights are typically set based on the image quality or importance of each view, with higher-quality images being given greater weights. The algorithm then performs a weighted summation of the same feature points from different viewpoints, fusing the feature information from multiple viewpoints through a weighted average to generate a composite feature map. This method effectively combines the advantages of multiple viewpoints, reducing noise and redundant information, thereby improving overall image quality and accuracy.

[0079] Based on the fused feature map, a structured light reconstruction algorithm is used to generate three-dimensional point cloud data;

[0080] It should be noted that the structured light reconstruction algorithm projects structured light of a known pattern onto the surface of an object, uses a camera to capture the deformation of the light pattern on the surface of the object, and combines geometric calculations to reconstruct the three-dimensional coordinates of the object, thereby generating accurate three-dimensional point cloud data.

[0081] Based on the three-dimensional point cloud data, a surface reconstruction algorithm is used to convert the point cloud data into a three-dimensional feature map of the substrate.

[0082] It should be noted that the surface reconstruction algorithm generates a continuous three-dimensional model by performing surface fitting on discrete points in the three-dimensional point cloud data. Methods such as Delaunay triangulation and Poisson surface reconstruction are usually used to ensure that the point cloud data is smoothly connected, and finally form a three-dimensional feature map of the substrate, which helps to more intuitively analyze the shape and defects of the substrate.

[0083] The defect probability prediction module is used to build a defect recognition model and predict the substrate defect probability based on the three-dimensional feature map of the substrate.

[0084] 3D-CNN is selected as the basic model. The 3D convolution kernel is used to extract local feature information from the 3D feature map of the substrate. The pooling layer is used to reduce the dimension of the 3D feature map of the substrate and retain important features. The fully connected layer is used for final classification to form the initial defect recognition model.

[0085] It should be noted that 3D-CNN was chosen as the base model because 3D convolutional neural networks (3D-CNNs) can directly process three-dimensional data, such as 3D feature maps or point cloud data, using 3D convolution kernels to simultaneously capture spatial feature information of depth, width, and height. This enables 3D-CNNs to extract more comprehensive spatial features than traditional CNNs when processing complex objects with spatial structures, such as package substrates, thereby improving the accuracy and precision of 3D defect recognition.

[0086] Based on historical 3D substrate data, the initial defect recognition model is trained using the stochastic gradient descent (SGD) optimization algorithm. The network parameters are adjusted to minimize the loss function to generate the final defect recognition model.

[0087] It should be noted that the model is trained using the stochastic gradient descent (SGD) optimization algorithm based on historical 3D substrate data. The specific process is as follows: First, the 3D substrate data is input into the model, and the error (loss function) between the output and the true label is calculated. Next, SGD uses the backpropagation algorithm to calculate the gradient of the loss function with respect to the model parameters. A small, randomly selected mini-batch of training data is then iteratively optimized, gradually adjusting the model parameters to reduce the loss function. This process is repeated until the model converges, ultimately generating a recognition model that can accurately predict substrate defects.

[0088] The three-dimensional feature map of the substrate is input into the defect recognition model to perform defect recognition on the substrate. The expression is:

[0089]

[0090] in, is the three-dimensional feature map of the substrate at the i-th viewing angle, is a nonlinear activation function, Represents the output features after being mapped by the feature mapping function at the i-th perspective, is a three-dimensional convolutional neural network, is the classification function, is the number of three-dimensional feature maps of the substrate, i is the feature index of the three-dimensional feature map of the substrate, represents the defect probability of the jth defect category.

[0091] It should be noted that by integrating 3D feature map information from multiple perspectives, a 3D convolutional neural network (3D-CNN) is used to deeply extract and analyze multidimensional spatial features. The nonlinear activation function σ and the feature mapping function ϕ ensure the nonlinear expression and effective mapping of complex features, enabling the model to accurately identify different types of substrate defects and predict the probability of each defect category. This process significantly improves the model's ability and accuracy in identifying a variety of complex defects, especially those that are hidden, tiny, or have multi-layer structures, significantly improving the system's detection accuracy and reliability.

[0092] The defect grade assessment module is used to predict the substrate defect grade score based on the substrate defect probability, assess the substrate defect grade based on the substrate defect grade score, and generate a substrate inspection report.

[0093] According to the probability of each defect Calculate the corresponding severity score. The severity of each defect category is nonlinearly mapped using the hyperbolic tangent function. The Gaussian kernel function is used to dynamically adjust the impact of each defect type on the total severity score of the substrate defect to obtain the final substrate defect severity score. The expression is:

[0094]

[0095] Where S is the severity score of the substrate, is the total number of defect categories, is the typical probability of defect type j, is the standard deviation of the probability of defect of type j, is the mapping slope control parameter of the j-th defect, is the mapping offset control parameter for the j-th type of defect.

[0096] It should be noted that It is obtained through statistical analysis of a large amount of historical inspection data. Specifically, the system collects and calculates the probability distribution of each defect category based on the defect identification results of previous packaging substrates. Then, these probability distributions are summarized and modeled, and the mean of the probability distribution of each defect category is taken as the typical probability of that type of defect. This typical probability reflects the frequency of occurrence of this defect class in historical data and its expected probability under normal circumstances.

[0097] It should also be noted that the severity of each defect category is nonlinearly mapped by the hyperbolic tangent function, and the probability of each defect is mapped by the Gaussian kernel function. Weighting is performed to dynamically adjust the impact of different defects on the overall score. This method not only accurately reflects the severity of each defect, but also fully considers the typical probabilities and standard deviations of different defects through weighted processing using a Gaussian kernel function, thereby avoiding the excessive impact of a single defect on the overall score. The beneficial effect of this method is that it can generate more accurate and balanced defect severity scores, effectively improving the inspection system's ability to comprehensively evaluate multiple defects, ensuring that the final score reflects the overall defect situation while highlighting the impact of key defect types.

[0098] Define the mild defect threshold s1 and severe risk threshold s2 based on historical data;

[0099] Furthermore, we first collect a large amount of substrate defect detection and scoring data, annotating and categorizing their severity. Then, by analyzing the score distribution, we typically define the 25th percentile as the mild defect threshold s1, indicating that defects below this value are relatively minor. The 90th or 95th percentile is defined as the severe risk threshold s2, indicating that defects above this value carry a greater risk and may cause substrate failure. Finally, we verify and adjust the thresholds based on actual test results to ensure their accuracy and rationality.

[0100] when When ≤s1, ​​the substrate defect level is considered to be a mild defect;

[0101] When s1< When <s2, the substrate defect level is considered to be moderate;

[0102] when When ≥s2, the substrate defect level is considered to be severe.

[0103] The substrate inspection report includes defect detection results, severity score, defect grade classification, three-dimensional feature map of the substrate, defect distribution statistics, substrate defect grade, and defect repair suggestions.

[0104] It should be noted that the defect detection results and severity scores are summarized, the three-dimensional feature map is saved, and the defect distribution statistical map is generated.

[0105] When using linear transformation to perform feature mapping on the input substrate 3D feature map at the i-th perspective, the feature mapping function The expression is:

[0106]

[0107] in, is the output feature after feature mapping at the i-th perspective, is the weight matrix of the i-th perspective, is the bias term of the i-th perspective;

[0108] The expression of the classification function f is:

[0109]

[0110] in, is the raw score output by the 3D-CNN output layer for the j-th defect type, is the probability of the j-th type of defect calculated by the Softmax function, It represents the original score output by the 3D-CNN output layer for the k-th defect type, where k is an index variable of the defect type used only in the exponential summation process.

[0111] It should be noted that and The raw scores generated by the 3D-CNN output layer for defects in category j and k are calculated through the forward propagation process. Specifically, after the input 3D feature map undergoes feature extraction through the 3D convolutional layer, pooling layer, and fully connected layer, the activation value of the final layer (usually the output of the fully connected layer) is the score for each category. These scores represent the 3D-CNN model's "confidence" in each defect category, that is, the model's tendency to believe that the input image belongs to a certain category. The Softmax function then converts these scores into a probability distribution, ensuring that the sum of the probabilities for each category is 1, for use in defect identification and classification.

[0112] In summary, the present invention achieves feature map registration and fusion based on multi-view images through: the feature map fusion module realizes feature map registration and fusion based on multi-view images, generates a three-dimensional feature map of the substrate, thereby capturing the spatial structure of the substrate and realizing accurate identification of complex spatial defects, overcoming the limitations of traditional two-dimensional detection; in addition, the defect probability prediction module constructs a defect recognition model based on a three-dimensional convolutional neural network (3D-CNN), and performs probability prediction on different defect types, solving the problem of single defect type identification in the existing technology, thereby realizing intelligent identification and evaluation of multiple defect types, and significantly improving the accuracy and automation level of detection.

[0113] Example 2, referring to Table 1, is the second example of the present invention. To further verify the technical solution of the present invention, experimental simulation data of an automated system for packaging substrate detection are provided.

[0114] This embodiment processes and analyzes substrate images collected from multiple perspectives by utilizing a data acquisition module, a feature extraction module, a feature map fusion module, a defect probability prediction module, etc., detects substrate defects, and compares the results with the prior art to demonstrate the advantages of the present invention.

[0115] First, the experiment used industrial cameras and 3D sensors mounted at different angles to capture multi-view image data of the substrate. The image data included the substrate's solder joints, edges, packaging layers, and surface. To ensure image quality, the captured image data was first filtered using a Gaussian filter to remove high-frequency noise while retaining key edge information. Next, an adaptive histogram equalization algorithm was used to enhance image contrast, making the details of the substrate image more clearly visible.

[0116] Next, feature extraction is performed on the preprocessed multi-view image data. The SIFT algorithm is used to detect key points and spatially align images from different perspectives, ensuring precise matching within the same coordinate system. Furthermore, the circular Hough transform algorithm is used to extract the geometric shape and texture features of the solder joints, the Canny edge detection algorithm to extract the contour features of the substrate edge, and the gray-level co-occurrence matrix (GLCM) algorithm to extract the texture features of the package layer.

[0117] Next, the feature maps are fused. The SIFT algorithm is used to match feature points in the same position in feature maps from different viewpoints. Affine transformation is then used to calculate the geometric transformation matrix, achieving precise alignment of the feature maps. Subsequently, a weighted average fusion algorithm is used to perform a weighted average fusion of the feature maps from multiple viewpoints, eliminating redundant information and generating a 3D feature map of the substrate, providing comprehensive feature information for subsequent defect identification.

[0118] Finally, based on the fused 3D feature map, a 3D-CNN model is used for defect identification. A 3D convolution kernel is used to extract local features of the substrate, and the probability of each defect category is calculated using a Softmax function. Subsequently, a defect probability prediction model is used to score the severity of each defect, ultimately generating a defect detection report for the substrate. Experimental results demonstrate that this method outperforms existing technologies in terms of solder joint identification accuracy, package layer texture extraction, edge detection integrity, and defect identification accuracy, demonstrating strong innovation and application value.

[0119] Existing technologies use traditional 2D image processing and feature extraction methods, including simple filtering and denoising, global histogram equalization enhancement, and edge detection based on pixel intensity changes. These technologies rely on 2D images to identify features such as solder joints, packaging layers, and edges, and lack multi-view image fusion and 3D feature processing. This results in low accuracy and robustness when identifying complex 3D structures or hidden defects, and they are unable to effectively process the spatial characteristics and multi-angle information of the substrate.

[0120] The details are shown in Table 1 below:

[0121] Table 1 Substrate defect detection experimental data

[0122] Parameter name Prior Art - Sample A Prior Art - Sample B Present invention - sample A Present invention - sample B Solder spot diameter deviation (%) 5.6 6.2 2.3 2.1 Solder point roundness error (%) 4.5 5.0 1.8 1.7 Encapsulation layer texture contrast 0.62 0.60 0.88 0.86 Edge detection completeness (%) 87.5 85.3 95.8 94.4 Defect recognition accuracy (%) 89.0 87.5 96.2 95.6

[0123] Through the analysis of the data in the above table, it can be clearly seen that the present invention is superior to the existing technology in multiple key performance indicators. The present invention achieves a higher defect recognition accuracy through multi-view image acquisition, precise feature extraction and three-dimensional feature fusion technology. For example, in terms of solder joint diameter deviation and roundness error, the error of the present invention is significantly lower than the existing technology, which is reduced by about 3 percentage points respectively, indicating that the extraction of the solder joint geometry is more accurate. In addition, the extraction contrast of the packaging layer texture is improved from 0.60 of the prior art to 0.88 of the present invention, indicating that the present invention has stronger expressiveness in the processing of surface details. The completeness of edge detection is also improved from 85% to more than 95%, reflecting the advantages of the present invention in substrate edge feature extraction. Overall, the innovations of the present invention in solder joint detection, edge recognition and packaging layer analysis have significantly improved the accuracy and reliability of defect detection.

[0124] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. An automated system for packaging substrate inspection, characterized in that: include, A data acquisition module is used to acquire multi-view image data and pre-process the multi-view image data; A feature extraction module is used to extract features from the preprocessed multi-view image data; A feature map fusion module is used to register and fuse the feature maps based on the extracted feature maps through an image registration algorithm to form a three-dimensional feature map of the substrate; Defect probability prediction module, used to build a defect recognition model and predict the substrate defect probability based on the three-dimensional feature map of the substrate; A defect grade assessment module is used to predict the substrate defect grade based on the substrate defect probability, assess the substrate defect grade based on the substrate defect grade score, and generate a substrate inspection report; The defect recognition model is constructed to predict the probability of substrate defects based on the three-dimensional feature map of the substrate. The specific steps are as follows: 3D-CNN is selected as the basic model. The 3D convolution kernel is used to extract local feature information from the 3D feature map of the substrate. The pooling layer is used to reduce the dimension of the 3D feature map of the substrate and retain important features. The fully connected layer is used for final classification to form the initial defect recognition model. Based on historical 3D substrate data, the initial defect recognition model is trained using a stochastic gradient descent optimization algorithm. The network parameters are adjusted to minimize the loss function to generate the final defect recognition model. The three-dimensional feature map of the substrate is input into the defect recognition model to perform defect recognition on the substrate. The expression is: in, is the three-dimensional feature map of the substrate at the i-th viewing angle, is a nonlinear activation function, Represents the output features after being mapped by the feature mapping function at the i-th perspective, is a three-dimensional convolutional neural network, is the classification function, is the number of three-dimensional feature maps of the substrate, i is the feature index of the three-dimensional feature map of the substrate, represents the defect probability of the jth defect category, where j is the index variable of the defect type; When using linear transformation to perform feature mapping on the input substrate 3D feature map at the i-th perspective, the feature mapping function The expression is: in, is the output feature after feature mapping at the i-th perspective, is the weight matrix of the i-th perspective, is the bias term of the i-th perspective; The expression of the classification function f is: in, is the raw score output by the 3D-CNN output layer for the j-th defect type, is the probability of the j-th type of defect calculated by the Softmax function, It represents the original score output by the 3D-CNN output layer for the k-th defect type. k is the index variable of the defect type used only in the exponential summation process. is the total number of defect categories.

2. The automated system for packaging substrate inspection according to claim 1, wherein: The multi-view image data includes image data of solder joints, edges, packaging layers, and surfaces of the substrate.

3. The automated system for packaging substrate inspection according to claim 2, wherein: The specific steps of preprocessing the multi-view image data are as follows: The collected multi-view image data is subjected to noise filtering through Gaussian filters; Enhance image contrast through adaptive histogram equalization; Use SIFT to spatially align images from different perspectives; The effective area of ​​the substrate is separated from the background through the edge detection algorithm.

4. The automated system for packaging substrate inspection according to claim 3, wherein: The feature extraction of the pre-processed multi-view image data is performed in the following specific steps: The circular Hough transform algorithm is used to identify the solder joint area in the image and extract the geometric shape feature map and texture feature map of the solder joint; The contour feature map of the substrate edge is extracted using the Canny edge detection algorithm; GLCM is used to extract the texture feature map of the substrate packaging layer area.

5. The automated system for packaging substrate inspection according to claim 4, wherein: Based on the extracted feature map, the feature map is registered and fused through the image registration algorithm to form a three-dimensional feature map of the substrate. The specific steps are as follows: Based on the extracted geometric shape and texture feature maps of the solder joints, the contour feature maps of the substrate edge, and the texture feature maps of the package area, SIFT is used to match feature points at the same position in the feature maps from different viewpoints. Use affine transformation to calculate the geometric transformation matrix between adjacent perspectives and accurately align feature maps from different perspectives; A weighted average fusion algorithm is used to fuse the feature information of the same feature map at different viewing angles, and an occlusion detection algorithm is used to identify blind spots at each viewing angle to eliminate redundant parts between different viewing angles. Based on the fused feature map, a structured light reconstruction algorithm is used to generate three-dimensional point cloud data; Based on the three-dimensional point cloud data, a surface reconstruction algorithm is used to convert the point cloud data into a three-dimensional feature map of the substrate.

6. The automated system for packaging substrate inspection according to claim 5, wherein: The specific steps of predicting the substrate defect degree score based on the substrate defect probability are as follows: According to the probability of each defect Calculate the corresponding severity score. The severity of each defect category is nonlinearly mapped using the hyperbolic tangent function. The Gaussian kernel function is used to dynamically adjust the impact of each defect type on the total severity score of the substrate defect to obtain the final substrate defect severity score. The expression is: in, is the severity score of the substrate, is the typical probability of defect type j, is the standard deviation of the probability of defect of type j, is the mapping slope control parameter of the j-th defect, is the mapping offset control parameter for the j-th type of defect.

7. The automated system for packaging substrate inspection according to claim 6, wherein: The specific steps of evaluating the substrate defect level according to the substrate defect degree score are as follows: Define the mild defect threshold s1 and severe risk threshold s2 based on historical data; when When ≤s1, ​​the substrate defect level is considered to be a mild defect; When s1< When <s2, the substrate defect level is considered to be moderate; when When ≥s2, the substrate defect level is considered to be severe.

8. The automated system for packaging substrate inspection according to claim 7, wherein: The substrate inspection report includes defect detection results, severity scores, defect grade classification, a three-dimensional feature map of the substrate, a defect distribution statistical map, substrate defect grade, and defect repair suggestions.

Images