Automatic system for packaging substrate detection

Abstract

The invention discloses an automatic system for packaging substrate detection, and relates to the technical field of data encryption, and the system comprises a data collection module which is used for collecting multi-view image data and carrying out the preprocessing of the multi-view image data; the feature extraction module is used for performing feature extraction on the preprocessed multi-view image data; the feature map fusion module is used for carrying out registration fusion on the feature maps through an image registration algorithm based on the extracted feature maps to form a three-dimensional feature map of the substrate; the defect probability prediction module is used for constructing a defect identification model and predicting the defect probability of the substrate based on the three-dimensional feature map of the substrate; and the defect grade evaluation module is used for predicting a substrate defect degree score according to the substrate defect probability, evaluating a substrate defect grade according to the substrate defect degree score, and generating a substrate detection report. According to the method, intelligent identification and evaluation of multiple defect types are realized, and the detection precision and the automation level are remarkably improved.

Description

Technical Field

[0001] The invention relates to the technical field of packaging substrate detection, in particular to an automated system for packaging substrate detection. Background Art

[0002] With the continuous improvement of the integration of electronic products, the quality and reliability of packaging substrates directly affect the performance and life of products. Therefore, how to efficiently and accurately detect packaging substrates has become an important research topic in the field of electronic manufacturing. Traditional packaging substrate detection methods mainly rely on artificial vision or optical detection equipment, which usually have problems such as slow detection speed, insufficient accuracy, and difficulty in identifying complex defects. In recent years, with the development of machine vision and deep learning technology, automated substrate detection systems have gradually become a research hotspot. Through the combination of multi-view image acquisition, feature extraction and intelligent algorithms, the automated detection system can complete a comprehensive inspection of the substrate surface and internal structure in a short time, and has the ability to identify and evaluate various complex defects (such as poor solder joints, packaging layer damage, etc.), thereby significantly improving detection efficiency and accuracy.

[0003] However, the existing automated substrate inspection systems still have some shortcomings in practical applications. First, most of the existing systems rely only on two-dimensional images for inspection, and lack in-depth analysis of the three-dimensional structure of the substrate, which makes it difficult to accurately identify some complex spatial defects (such as hidden defects in multi-layer structures). Secondly, the existing defect recognition methods often only deal with a single defect type and lack the comprehensive ability to uniformly evaluate multiple defect types. In particular, the accuracy and reliability of existing methods need to be improved in the quantitative assessment of the severity of defects. Therefore, how to construct a three-dimensional feature map of the substrate through multi-view image fusion technology, and based on the three-dimensional feature map, to achieve accurate identification and severity assessment of multiple defects has become an urgent problem to be solved in the field of automated inspection. Summary of the invention

[0004] In view of the above existing problems, the present invention is proposed.

[0005] Therefore, the present invention provides an automated system for packaging substrate inspection to solve the problems in the prior art that two-dimensional inspection cannot accurately identify complex spatial defects and the defect severity assessment is inaccurate.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0007] In a first aspect, the present invention provides an automated system for packaging substrate inspection, comprising: a data acquisition module for acquiring multi-view image data and preprocessing the multi-view image data; a feature extraction module for extracting features from the preprocessed multi-view image data; a feature map fusion module for aligning and fusing the feature maps based on the extracted feature maps through an image alignment algorithm to form a three-dimensional feature map of the substrate; a defect probability prediction module for constructing a defect recognition model to predict the substrate defect probability based on the three-dimensional feature map of the substrate; a defect level assessment module for predicting a substrate defect degree score based on the substrate defect probability, assessing the substrate defect level based on the substrate defect degree score, and generating a substrate inspection report.

[0008] As a preferred solution of the automated system for packaging substrate detection of the present invention, the multi-view image data includes image data of the solder joints, edges, packaging layers and surfaces of the substrate.

[0009] As a preferred solution of the automated system for packaging substrate inspection of the present invention, the preprocessing of the multi-view image data is performed in the following specific steps:

[0010] The collected multi-view image data is subjected to noise filtering through a Gaussian filter;

[0011] Enhance image contrast through adaptive histogram equalization;

[0012] Use SIFT to spatially align images from different perspectives;

[0013] The effective area of ​​the substrate is separated from the background through the edge detection algorithm.

[0014] As a preferred solution of the automated system for packaging substrate detection of the present invention, the following specific steps are performed to extract features from the pre-processed multi-view image data:

[0015] The circular Hough transform algorithm is used to identify the solder joint area in the image and extract the geometric shape feature map and texture feature map of the solder joint;

[0016] The contour feature map of the substrate edge is extracted by using the Canny edge detection algorithm;

[0017] GLCM is used to extract the texture feature map of the substrate packaging layer area.

[0018] As a preferred solution of the automated system for packaging substrate detection described in the present invention, wherein: based on the extracted feature map, the feature map is registered and fused through an image registration algorithm to form a three-dimensional feature map of the substrate. The specific steps are as follows:

[0019] Based on the extracted geometry and texture feature maps of solder joints, the contour feature maps of substrate edges, and the texture feature maps of package areas, SIFT is used to match feature points at the same position in feature maps of different viewing angles.

[0020] Use affine transformation to calculate the geometric transformation matrix between adjacent perspectives and accurately align feature maps from different perspectives;

[0021] The weighted average fusion algorithm is used to fuse the feature information of the same feature map at different viewing angles, and the occlusion detection algorithm is used to identify the blind spot areas at each viewing angle to eliminate the redundant parts between different viewing angles.

[0022] Based on the fused feature map, a structured light reconstruction algorithm is used to generate three-dimensional point cloud data;

[0023] Based on the three-dimensional point cloud data, a surface reconstruction algorithm is used to convert the point cloud data into a three-dimensional feature map of the substrate.

[0024] As a preferred solution of the automated system for packaging substrate detection of the present invention, wherein: the defect recognition model is constructed to predict the probability of substrate defects based on the three-dimensional feature map of the substrate, and the specific steps are as follows:

[0025] 3D-CNN is selected as the basic model. The local feature information in the three-dimensional feature map of the substrate is extracted through the three-dimensional convolution kernel. The three-dimensional feature map of the substrate is reduced in dimension and important features are retained through the pooling layer. The fully connected layer is used for the final classification to form the initial defect recognition model.

[0026] Based on historical 3D substrate data, the initial defect recognition model is trained using a stochastic gradient descent optimization algorithm, and the network parameters are adjusted to minimize the loss function to generate the final defect recognition model.

[0027] The three-dimensional feature map of the substrate is input into the defect recognition model to identify the defects of the substrate. The expression is:

[0028]

[0029] Among them, T i is the three-dimensional feature map of the substrate at the i-th viewing angle, σ is the nonlinear activation function, φ(T i ) represents the output feature after being mapped by the feature mapping function at the i-th perspective, D 3D-CNN is a three-dimensional convolutional neural network, f is a classification function, n is the number of three-dimensional feature maps of the substrate, i is the feature index of the three-dimensional feature map of the substrate, P j represents the defect probability of the jth defect category, where j is the index variable of the defect type.

[0030] As a preferred solution of the automated system for packaging substrate detection of the present invention, wherein: the substrate defect degree score is predicted according to the substrate defect probability, and the specific steps are as follows:

[0031] According to the probability P of each defect j Calculate the corresponding severity score. The severity of each defect category is nonlinearly mapped through the hyperbolic tangent function, and the Gaussian kernel function is used to dynamically adjust the impact of various defects on the total severity score of the substrate defect to obtain the final substrate defect severity score. The expression is:

[0032]

[0033] Where S is the severity score of the substrate, m is the total number of defect categories, and μ j is the typical probability of the jth type of defect, δ j is the standard deviation of the probability of defect of type j, a j is the mapping slope control parameter of the j-th defect, b j is the mapping offset control parameter of the j-th defect.

[0034] As a preferred solution of the automated system for packaging substrate detection of the present invention, wherein: the substrate defect level is evaluated according to the substrate defect level score, and the specific steps are as follows:

[0035] Define the minor defect threshold s1 and the severe risk threshold s2 based on historical data;

[0036] When S≤s1, the substrate defect level is considered to be a mild defect;

[0037] When s1<S<s2, the substrate defect level is considered to be moderate;

[0038] When S≥s2, the substrate defect level is considered to be severe.

[0039] As a preferred solution of the automated system for packaging substrate inspection described in the present invention, the substrate inspection report includes defect detection results, severity scores, defect level classification, three-dimensional feature maps of the substrate, defect distribution statistics, substrate defect levels, and defect repair suggestions.

[0040] As a preferred solution of the automated system for packaging substrate detection of the present invention, when a linear transformation is used to perform feature mapping on the input three-dimensional feature map of the substrate at the i-th viewing angle, the feature mapping function φ is expressed as:

[0041] φ(T i )=W i ·T i +g i;

[0042] Among them, φ(T i ) is the output feature after feature mapping at the i-th perspective, W i is the weight matrix of the i-th view, g i is the bias term of the i-th perspective;

[0043] The expression of the classification function f is:

[0044]

[0045] Among them, z j is the raw score output by the 3D-CNN output layer for the j-th defect type, f(z j ) is the probability of the j-th defect calculated by the Softmax function, z k Represents the original score output by the 3D-CNN output layer for the k-th defect type, where k is an index variable of the defect type used only in the exponential summation process.

[0046] The beneficial effects of the present invention are as follows: through the feature map fusion module, feature map registration and fusion based on multi-view images are realized, and a three-dimensional feature map of the substrate is generated, so that the spatial structure of the substrate can be captured, and accurate recognition of complex spatial defects can be achieved, overcoming the limitations of traditional two-dimensional detection; in addition, the defect probability prediction module constructs a defect recognition model based on a three-dimensional convolutional neural network (3D-CNN), and performs probability prediction on different defect types, solving the problem of single defect type recognition in the prior art, thereby realizing intelligent recognition and evaluation of multiple defect types, and significantly improving the detection accuracy and automation level. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0048] Figure 1 Schematic diagram of the automated system for packaging substrate inspection in Example 1.

[0049] Figure 2 This is a schematic diagram of preprocessing multi-view image data in Example 1. DETAILED DESCRIPTION

[0050] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings.

[0051] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0052] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive with other embodiments.

[0053] Example 1, reference Figure 1 and Figure 2 , which is the first embodiment of the present invention, provides an automated system for packaging substrate detection, comprising:

[0054] The data acquisition module is used to acquire multi-view image data and pre-process the multi-view image data.

[0055] The multi-view image data includes image data of solder joints, edges, packaging layers and surfaces of the substrate;

[0056] Furthermore, the above data are collected by industrial cameras or 3D sensors installed at different angles;

[0057] The collected multi-view image data is subjected to noise filtering through a Gaussian filter;

[0058] For example, a Gaussian filter uses a weighted average to smooth an image, remove high-frequency noise, and retain the main edge information. A Gaussian filter uses a weighted average to smooth an image, remove high-frequency noise, and retain the main edge information.

[0059] Enhance image contrast through adaptive histogram equalization;

[0060] For example, adaptive histogram equalization adjusts the image brightness distribution according to the local area, enhancing contrast and detail visibility.

[0061] Use SIFT to spatially align images from different perspectives;

[0062] For example, the SIFT algorithm is used to detect and match key points in an image, calculate the geometric transformation matrix, and achieve accurate alignment of multi-view images.

[0063] The effective area of ​​the substrate is separated from the background through the edge detection algorithm.

[0064] For example, significant edges in the image are determined by edge detection algorithms, thereby isolating the effective area of ​​the substrate and eliminating background interference.

[0065] The feature extraction module is used to extract features from the preprocessed multi-view image data.

[0066] The circular Hough transform algorithm is used to identify the solder joint area in the image and extract the geometric shape feature map and texture feature map of the solder joint;

[0067] Furthermore, after using the circular Hough transform algorithm to identify the solder joint area, the geometric shape features (such as diameter and roundness) and texture features (such as surface roughness and texture direction) of the solder joint are extracted. By combining the geometric and texture information, more accurate and rich feature support is provided for the comprehensive evaluation of solder joint quality and defect detection.

[0068] The contour feature map of the substrate edge is extracted by using the Canny edge detection algorithm;

[0069] Furthermore, through the Canny edge detection algorithm, the image is first denoised by Gaussian filtering, then the image gradient is calculated and the edge is refined using the non-maximum suppression method, then the strong and weak edges are screened by double thresholds, and finally the complete contour feature map of the substrate is generated by edge connection, providing accurate edge information for subsequent detection.

[0070] GLCM is used to extract the texture feature map of the substrate packaging layer area.

[0071] Furthermore, the texture feature map of the substrate packaging layer area is extracted through the gray-level co-occurrence matrix (GLCM). First, the packaging layer image is grayed, and then the co-occurrence frequency of pixel pairs with different gray values ​​is calculated. Then, texture features such as contrast, energy, and correlation are extracted from it to help describe the roughness and consistency of the packaging layer, providing important texture information for subsequent defect detection.

[0072] The feature map fusion module is used to align and fuse the feature maps based on the extracted feature maps through an image registration algorithm to form a three-dimensional feature map of the substrate.

[0073] Based on the extracted geometry and texture feature maps of solder joints, the contour feature maps of substrate edges, and the texture feature maps of package areas, SIFT is used to match feature points at the same position in feature maps of different viewing angles.

[0074] Furthermore, key points are detected in feature maps of different perspectives, and feature descriptors that are not affected by scaling, rotation, and illumination changes are generated for each key point; then, the Euclidean distance between these descriptors is calculated to find matching feature points at the same position in images of different perspectives; finally, the optimal matching points are screened out by methods such as nearest neighbor matching or ratio testing.

[0075] Use affine transformation to calculate the geometric transformation matrix between adjacent perspectives and accurately align feature maps from different perspectives;

[0076] Furthermore, through affine transformation, the geometric transformation matrix (including rotation, translation and scaling) between adjacent perspectives is calculated using the matched feature point pairs, and the matrix is ​​applied to accurately align the feature maps of different perspectives to the same coordinate system.

[0077] The weighted average fusion algorithm is used to fuse the feature information of the same feature map at different viewing angles, and the occlusion detection algorithm is used to identify the blind spot areas at each viewing angle to eliminate the redundant parts between different viewing angles.

[0078] It should be noted that the weighted average fusion algorithm assigns different weights to the feature map of each view. These weights are usually set according to the image quality or importance of each view, and higher quality images are given greater weights. Then, the same feature point under different viewpoints is weighted and summed, and the feature information of multiple viewpoints is fused by weighted average to generate a comprehensive feature map. This method can effectively combine the advantages of multiple viewpoints, reduce noise and redundant information, and thus improve the quality and accuracy of the overall image.

[0079] Based on the fused feature map, a structured light reconstruction algorithm is used to generate three-dimensional point cloud data;

[0080] It should be noted that the structured light reconstruction algorithm projects structured light of a known pattern onto the surface of an object, uses a camera to capture the deformation of the light pattern on the surface of the object, and combines geometric calculations to reconstruct the three-dimensional coordinates of the object, thereby generating accurate three-dimensional point cloud data.

[0081] Based on the three-dimensional point cloud data, a surface reconstruction algorithm is used to convert the point cloud data into a three-dimensional feature map of the substrate.

[0082] It should be noted that the surface reconstruction algorithm generates a continuous three-dimensional model by fitting the discrete points in the three-dimensional point cloud data into a surface. Methods such as Delaunay triangulation and Poisson surface reconstruction are usually used to ensure that the point cloud data is smoothly connected, and finally form a three-dimensional feature map of the substrate, which helps to analyze the shape and defects of the substrate more intuitively.

[0083] The defect probability prediction module is used to build a defect recognition model and predict the substrate defect probability based on the three-dimensional feature map of the substrate.

[0084] 3D-CNN is selected as the basic model. The local feature information in the three-dimensional feature map of the substrate is extracted through the three-dimensional convolution kernel. The three-dimensional feature map of the substrate is reduced in dimension and important features are retained through the pooling layer. The fully connected layer is used for the final classification to form the initial defect recognition model.

[0085] It should be noted that the reason for choosing 3D-CNN as the basic model is that 3D convolutional neural network (3D-CNN) can directly process three-dimensional data, such as three-dimensional feature maps or point cloud data, and simultaneously capture the feature information of depth, width and height in space through three-dimensional convolution kernels. This enables 3D-CNN to extract spatial features more comprehensively than traditional CNN when processing complex objects with spatial structures (such as packaging substrates), thereby improving the accuracy and precision of identifying three-dimensional defects.

[0086] Based on historical 3D substrate data, the initial defect recognition model is trained using the stochastic gradient descent (SGD) optimization algorithm, and the network parameters are adjusted to minimize the loss function to generate the final defect recognition model.

[0087] It should be noted that the model is trained using the stochastic gradient descent (SGD) optimization algorithm based on historical 3D substrate data. The specific process is: first, the 3D substrate data is input into the model, and the error (loss function) between the output result and the true label is calculated. Then, SGD calculates the gradient of the loss function relative to the model parameters through the back-propagation algorithm, and randomly extracts a small part of the training data (mini-batch) for iterative optimization, gradually adjusting the model parameters to reduce the value of the loss function. This process is repeated until the model converges, and finally generates a recognition model that can accurately predict substrate defects.

[0088] The three-dimensional feature map of the substrate is input into the defect recognition model to identify the defects of the substrate. The expression is:

[0089]

[0090] Among them, T i is the three-dimensional feature map of the substrate at the i-th viewing angle, σ is the nonlinear activation function, φ(T i ) represents the output feature after being mapped by the feature mapping function at the i-th perspective, D 3D-CNN is a three-dimensional convolutional neural network, f is a classification function, n is the number of three-dimensional feature maps of the substrate, i is the feature index of the three-dimensional feature map of the substrate, P j represents the defect probability of the jth defect category.

[0091] It should be noted that the three-dimensional feature map information from multiple perspectives is integrated, and the multi-dimensional spatial features are deeply extracted and analyzed with the help of a three-dimensional convolutional neural network (3D-CNN). The nonlinear activation function σ and the feature mapping function φ ensure the nonlinear expression and effective mapping of complex features, so that the model can accurately identify different types of substrate defects and predict the probability P of each defect category. jThis process greatly improves the model's ability and accuracy in identifying a variety of complex defects, especially hidden, tiny or multi-layered defects, and significantly improves the system's detection accuracy and reliability.

[0092] The defect grade assessment module is used to predict the substrate defect grade score according to the substrate defect probability, assess the substrate defect grade according to the substrate defect grade score, and generate a substrate inspection report.

[0093] According to the probability P of each defect j Calculate the corresponding severity score. The severity of each defect category is nonlinearly mapped through the hyperbolic tangent function, and the Gaussian kernel function is used to dynamically adjust the impact of various defects on the total severity score of the substrate defect to obtain the final substrate defect severity score. The expression is:

[0094]

[0095] Where S is the severity score of the substrate, m is the total number of defect categories, and μ j is the typical probability of the jth type of defect, δ j is the standard deviation of the probability of defect of type j, a j is the mapping slope control parameter of the j-th defect, b j is the mapping offset control parameter of the j-th defect.

[0096] It should be noted that μ j It is obtained through statistical analysis of a large amount of historical inspection data. Specifically, the system collects and calculates the probability distribution of each defect category based on the defect recognition results of previous packaging substrates. Then, these probability distributions are summarized and modeled, and the mean of the probability distribution of each defect category is taken as the typical probability μ of this type of defect. j This typical probability reflects the frequency of occurrence of this defect class in historical data and its expected probability under normal circumstances.

[0097] It should also be noted that the severity of each defect type is nonlinearly mapped by the hyperbolic tangent function, and the probability P of each defect is calculated by combining the Gaussian kernel function. j Weighting is performed to dynamically adjust the impact of different defects on the total score. This method can not only accurately reflect the severity of each defect, but also fully consider the typical probability and standard deviation of different defects through weighted processing of Gaussian kernel function, thereby avoiding the excessive impact of a single defect on the total score. The beneficial effect of this method is that it can generate more accurate and balanced defect severity scores, effectively improve the comprehensive evaluation ability of the detection system for multiple defects, and ensure that the final score reflects both the global defect situation and the impact of key defect types.

[0098] Define the minor defect threshold s1 and the severe risk threshold s2 based on historical data;

[0099] Furthermore, first, a large amount of substrate defect detection and scoring data is collected, and their severity is annotated and classified. Then, by analyzing the distribution of the scores, the 25% percentile of the score is usually defined as the mild defect threshold s1, indicating that defects below this value are mild. The 90% or 95% percentile is defined as the severe risk threshold s2, indicating that defects above this value have a greater risk and may cause substrate failure. Finally, verification and adjustment are carried out through actual test results to ensure the accuracy and rationality of the threshold.

[0100] When S≤s1, the substrate defect level is considered to be a mild defect;

[0101] When s1<S<s2, the substrate defect level is considered to be moderate;

[0102] When S≥s2, the substrate defect level is considered to be severe.

[0103] The substrate inspection report includes defect detection results, severity scores, defect grade classification, three-dimensional feature map of the substrate, defect distribution statistics and substrate defect grade, as well as defect repair suggestions.

[0104] It should be noted that the defect detection results and severity scores are summarized, the three-dimensional feature map is saved, and the defect distribution statistical map is generated.

[0105] When a linear transformation is used to perform feature mapping on the three-dimensional feature map of the input substrate at the i-th viewing angle, the feature mapping function φ is expressed as:

[0106] φ(T i )=W i ·T i +g i ;

[0107] Among them, φ(T i ) is the output feature after feature mapping at the i-th perspective, W i is the weight matrix of the i-th view, g i is the bias term of the i-th perspective;

[0108] The expression of the classification function f is:

[0109]

[0110] Among them, z j is the raw score output by the 3D-CNN output layer for the j-th defect type, f(z j ) is the probability of the j-th defect calculated by the Softmax function, z kRepresents the original score output by the 3D-CNN output layer for the k-th defect type, where k is an index variable of the defect type used only in the exponential summation process.

[0111] It should be noted that z j and z k It is the original score generated by the 3D-CNN output layer for the j-th and k-th defects, which is calculated through the forward propagation process. Specifically, after the input 3D feature map is extracted by the 3D convolution layer, pooling layer and fully connected layer, the activation value of the last layer (usually the output of the fully connected layer) is the score of each category. These scores represent the "confidence" of the 3D-CNN model in each defect category, that is, the tendency of the model to believe that the input image belongs to a certain category. Then, the Softmax function converts these scores into probability distributions based on them, ensuring that the sum of the probabilities of each category is 1, which is used for defect recognition and classification.

[0112] In summary, the present invention achieves feature map registration and fusion based on multi-view images through: the feature map fusion module realizes feature map registration and fusion based on multi-view images, generates a three-dimensional feature map of the substrate, thereby capturing the spatial structure of the substrate, and realizing accurate recognition of complex spatial defects, thus overcoming the limitations of traditional two-dimensional detection; in addition, the defect probability prediction module constructs a defect recognition model based on a three-dimensional convolutional neural network (3D-CNN), and performs probability prediction on different defect types, thus solving the problem of single defect type recognition in the prior art, thereby realizing intelligent recognition and evaluation of multiple defect types, and significantly improving the detection accuracy and automation level.

[0113] Example 2, referring to Table 1, is the second example of the present invention. To further verify the technical solution of the present invention, experimental simulation data of an automated system for packaging substrate detection are provided.

[0114] This embodiment processes and analyzes substrate images collected from multiple perspectives by using a data acquisition module, a feature extraction module, a feature map fusion module, a defect probability prediction module, etc., detects defects in the substrate, and compares it with the prior art to demonstrate the advantages of the present invention.

[0115] First, the experiment uses industrial cameras and 3D sensors installed at different angles to collect multi-view image data of the substrate, including the solder joints, edges, packaging layers, and surfaces of the substrate. To ensure image quality, the collected image data is first filtered using a Gaussian filter to remove high-frequency noise and retain the main edge information. Then, an adaptive histogram equalization algorithm is used to enhance the image contrast, making the details of the substrate image more clearly visible.

[0116] Secondly, feature extraction is performed on the preprocessed multi-view image data. The SIFT algorithm is used to detect key points and spatially align images from different perspectives to ensure that images from different perspectives are accurately matched in the same coordinate system. In addition, the circular Hough transform algorithm is used to extract the geometric shape and texture features of the solder joints, the Canny edge detection algorithm is used to extract the contour features of the substrate edge, and the gray level co-occurrence matrix (GLCM) algorithm is used to extract the texture features of the packaging layer.

[0117] Next, the feature map is fused. The SIFT algorithm is used to match the feature points at the same position in the feature maps of different viewpoints, and the geometric transformation matrix is ​​calculated using affine transformation to achieve accurate alignment of the feature maps. Subsequently, the weighted average fusion algorithm is used to perform weighted average fusion on the feature maps of multiple viewpoints to eliminate redundant information and generate a three-dimensional feature map of the substrate, providing comprehensive feature information for subsequent defect identification.

[0118] Finally, based on the fused three-dimensional feature map, the 3D-CNN model is used for defect recognition. The local features of the substrate are extracted by the three-dimensional convolution kernel, and the probability of each defect category is calculated by the Softmax function. Subsequently, the severity of each defect is scored using the defect probability prediction model, and finally a defect detection report for the substrate is generated. The experimental results show that the present invention is superior to the existing technology in terms of solder joint recognition accuracy, packaging layer texture extraction, edge detection integrity, and defect recognition accuracy, demonstrating strong innovation and application value.

[0119] The existing technology uses traditional two-dimensional image processing and feature extraction methods, including simple filtering and denoising, global histogram equalization enhancement, and edge detection methods based on pixel intensity changes. The existing technology relies on two-dimensional images to identify features such as solder joints, packaging layers, and edges, and lacks the fusion of multi-view images and three-dimensional feature processing. This results in low accuracy and robustness of the existing technology when identifying complex three-dimensional structures or hidden defects, and it cannot effectively process the spatial features and multi-angle information of the substrate.

[0120] The details are shown in Table 1 below:

[0121] Table 1 Substrate defect detection experimental data table

[0122]

[0123]

[0124] Through the analysis of the data in the above table, it can be clearly seen that the present invention is superior to the prior art in multiple key performance indicators. The present invention achieves a higher defect recognition accuracy through multi-view image acquisition, precise feature extraction and three-dimensional feature fusion technology. For example, in terms of solder joint diameter deviation and roundness error, the error of the present invention is significantly lower than that of the prior art, which is reduced by about 3 percentage points respectively, indicating that the extraction of solder joint geometry is more accurate. In addition, the extraction contrast of the packaging layer texture is improved from 0.60 of the prior art to 0.88 of the present invention, indicating that the present invention has stronger expressiveness in the processing of surface details. The completeness of edge detection is also improved from 85% to more than 95%, reflecting the advantages of the present invention in substrate edge feature extraction. Overall, the innovations of the present invention in solder joint detection, edge recognition and packaging layer analysis have significantly improved the accuracy and reliability of defect detection.

[0125] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. An automated system for packaging substrate inspection, characterized in that: include, A data acquisition module, used for acquiring multi-view image data and preprocessing the multi-view image data; A feature extraction module is used to extract features from the preprocessed multi-view image data; A feature map fusion module is used to register and fuse the feature maps based on the extracted feature maps through an image registration algorithm to form a three-dimensional feature map of the substrate; The defect probability prediction module is used to build a defect recognition model and predict the defect probability of the substrate based on the three-dimensional feature map of the substrate; The defect grade assessment module is used to predict the substrate defect grade score according to the substrate defect probability, assess the substrate defect grade according to the substrate defect grade score, and generate a substrate inspection report.

2. The automated system for packaging substrate inspection according to claim 1, wherein: The multi-view image data includes image data of solder joints, edges, packaging layers and surfaces of the substrate.

3. The automated system for packaging substrate inspection as claimed in claim 2, characterized in that: The specific steps of preprocessing the multi-view image data are as follows: The collected multi-view image data is subjected to noise filtering through a Gaussian filter; Enhance image contrast through adaptive histogram equalization; Use SIFT to spatially align images from different perspectives; The effective area of ​​the substrate is separated from the background through the edge detection algorithm.

4. The automated system for packaging substrate inspection as claimed in claim 3, characterized in that: The feature extraction of the preprocessed multi-view image data is performed in the following specific steps: The circular Hough transform algorithm is used to identify the solder joint area in the image and extract the geometric shape feature map and texture feature map of the solder joint; The contour feature map of the substrate edge is extracted by using the Canny edge detection algorithm; GLCM is used to extract the texture feature map of the substrate packaging layer area.

5. The automated system for packaging substrate inspection according to claim 4, characterized in that: Based on the extracted feature map, the feature map is registered and fused through an image registration algorithm to form a three-dimensional feature map of the substrate. The specific steps are as follows: Based on the extracted geometry and texture feature maps of solder joints, the contour feature maps of substrate edges, and the texture feature maps of package areas, SIFT is used to match feature points at the same position in feature maps of different viewing angles. Use affine transformation to calculate the geometric transformation matrix between adjacent perspectives and accurately align feature maps from different perspectives; The weighted average fusion algorithm is used to fuse the feature information of the same feature map at different viewing angles, and the occlusion detection algorithm is used to identify the blind spot areas at each viewing angle to eliminate the redundant parts between different viewing angles. Based on the fused feature map, a structured light reconstruction algorithm is used to generate three-dimensional point cloud data; Based on the three-dimensional point cloud data, a surface reconstruction algorithm is used to convert the point cloud data into a three-dimensional feature map of the substrate.

6. The automated system for packaging substrate inspection according to claim 5, characterized in that: The defect recognition model is constructed to predict the probability of substrate defects based on the three-dimensional feature map of the substrate. The specific steps are as follows: 3D-CNN is selected as the basic model. The local feature information in the three-dimensional feature map of the substrate is extracted through the three-dimensional convolution kernel. The three-dimensional feature map of the substrate is reduced in dimension and important features are retained through the pooling layer. The fully connected layer is used for the final classification to form the initial defect recognition model. Based on historical 3D substrate data, the initial defect recognition model is trained using a stochastic gradient descent optimization algorithm, and the network parameters are adjusted to minimize the loss function to generate the final defect recognition model. The three-dimensional feature map of the substrate is input into the defect recognition model to identify the defects of the substrate. The expression is: Among them, T i is the three-dimensional feature map of the substrate at the i-th viewing angle, σ is the nonlinear activation function, φ(T i ) represents the output feature after being mapped by the feature mapping function at the i-th perspective, D 3D-CNN is a three-dimensional convolutional neural network, f is a classification function, n is the number of three-dimensional feature maps of the substrate, i is the feature index of the three-dimensional feature map of the substrate, P j represents the defect probability of the jth defect category, where j is the index variable of the defect type.

7. The automated system for packaging substrate inspection according to claim 6, wherein: The specific steps of predicting the substrate defect level score according to the substrate defect probability are as follows: According to the probability P of each defect j Calculate the corresponding severity score. The severity of each defect category is nonlinearly mapped through the hyperbolic tangent function, and the Gaussian kernel function is used to dynamically adjust the impact of various defects on the total severity score of the substrate defect to obtain the final substrate defect severity score. The expression is: Where S is the severity score of the substrate, m is the total number of defect categories, and μ j is the typical probability of the jth type of defect, δ j is the standard deviation of the probability of defect of type j, a j is the mapping slope control parameter of the j-th defect, b j is the mapping offset control parameter of the j-th defect.

8. The automated system for packaging substrate inspection according to claim 7, wherein: The specific steps of evaluating the substrate defect level according to the substrate defect level score are as follows: Define the minor defect threshold s1 and the severe risk threshold s2 based on historical data; When S≤s1, the substrate defect level is considered to be a mild defect; When s1<S<s2, the substrate defect level is considered to be moderate; When S≥s2, the substrate defect level is considered to be severe.

9. The automated system for packaging substrate inspection according to claim 8, wherein: The substrate inspection report includes defect inspection results, severity scores, defect grade classification, three-dimensional feature maps of the substrate, defect distribution statistics, substrate defect grades, and defect repair suggestions.

10. The automated system for packaging substrate inspection according to claim 6, wherein: When a linear transformation is used to perform feature mapping on the three-dimensional feature map of the input substrate at the i-th viewing angle, the feature mapping function φ is expressed as: φ(T i )=W i ·T i +g i ; Among them, φ(T i ) is the output feature after feature mapping at the i-th perspective, W i is the weight matrix of the i-th view, g i is the bias term of the i-th perspective; The expression of the classification function f is: Among them, z j is the raw score output by the 3D-CNN output layer for the j-th defect type, f(z j ) is the probability of the j-th defect calculated by the Softmax function, z k Represents the original score output by the 3D-CNN output layer for the k-th defect type, where k is an index variable of the defect type used only in the exponential summation process.

Images