SMT welding spot defect detection method based on image data

Abstract

The invention discloses an SMT welding spot defect detection method based on image data. The method comprises the following steps: S1, acquiring an image of an SMT welding spot area; s2, carrying out image preprocessing; s3, carrying out welding spot region segmentation by adopting a self-adaptive region growing algorithm, and carrying out modeling on a welding spot region contour in combination with a boundary fitting optimization algorithm to generate a welding spot contour model; s4, multi-level comprehensive features are extracted from the welding spot area, and the multi-level comprehensive features are constructed into a feature vector set; and S5, inputting the feature vector set into a welding spot defect detection network, predicting a defect mode existing in the current batch of welding spots by adopting an improved ProtoNet model, and completing classification and identification of welding spot defects by adopting adaptive feature filtering. According to the method, the adaptive region growing algorithm and the improved ProtoNet model are adopted, accurate detection of the welding spot defects is achieved, and the method has the advantages of being high in detection precision, high in adaptability and good in robustness.

Description

Technical Field

[0001] The present invention relates to the field of surface mount technology, and in particular to an SMT solder joint defect detection method based on image data. Background Art

[0002] Surface mount technology is the core process in the field of modern electronic manufacturing. Its production efficiency and product quality are directly related to the performance and reliability of electronic devices. SMT solder joints are important structures connecting electronic components and printed circuit boards. Their welding quality has a vital impact on the functional integrity of the entire electronic product. Therefore, efficient and accurate detection of SMT solder joint defects has become an important research direction and practical demand in the electronic manufacturing industry.

[0003] Existing SMT solder joint defect detection methods can be mainly divided into manual visual inspection, rule-based image processing methods, and machine learning-based automated detection methods. Manual visual inspection, as the earliest detection method, relies on the operator's experience for visual judgment. It has problems such as high labor intensity, susceptibility to subjective interference, poor detection consistency, and high missed detection rate. It can no longer meet the high-density, high-precision, and high-rate production requirements of modern electronic manufacturing. Rule-based image processing methods usually rely on traditional visual algorithms such as preset thresholds, morphological operations, and edge detection to analyze the appearance characteristics of solder joints and then determine the defect type. However, because SMT solder joints are subject to multiple factors such as process fluctuations, material differences, and environmental changes during different batch production processes, solder joint images have large variability in brightness, contrast, texture characteristics, morphological contours, etc., making it difficult for fixed rules or simple models to adapt to complex and changing actual scenarios, resulting in false detections, missed detections, and insufficient detection accuracy.

[0004] In recent years, with the development of computer vision and artificial intelligence technologies, solder joint defect detection methods based on machine learning, especially deep learning, have been gradually studied and applied. For example, convolutional neural networks are used to classify solder joint images, and detection accuracy is improved through end-to-end training. These methods significantly improve feature representation capabilities compared to traditional image processing methods, capable of automatically extracting complex image features of solder joints and achieving relatively stable detection results. However, traditional deep learning methods still face many challenges when applied to SMT solder joint defect detection. First, the difficulty in collecting large-scale SMT solder joint defect samples and the high cost of data annotation lead to overfitting and limited generalization of deep learning models. Second, image differences caused by different PCB materials, pad morphologies, and soldering processes lead to significant performance degradation when the trained models are transferred to new batches and materials, requiring frequent retraining or parameter adjustment, which reduces the industrial applicability of the systems. Third, most existing deep learning-based methods use a uniform feature scale for feature extraction and lack specialized modeling for fine-grained solder joint features (such as local curvature changes and subtle texture anomalies), resulting in insufficient detection capabilities for subtle defects.

[0005] At the feature extraction level, traditional methods usually rely on global statistical features, such as the overall brightness, area, and perimeter of the solder joint, ignoring the comprehensive expression of the local morphological characteristics, boundary curvature change characteristics, and local texture structure characteristics of the solder joint, limiting the detection system's sensitivity to complex defects. In addition, existing methods often use fixed filters and simple normalization processing in the image preprocessing stage, and fail to make adaptive adjustments based on the characteristics of different solder joint areas. As a result, the preprocessing effect fluctuates greatly under different image quality conditions, affecting the stability of subsequent feature extraction and defect recognition.

[0006] In response to the above problems, the existing technology has not yet provided an SMT solder joint defect detection method that takes into account high adaptability, high detection accuracy and high industrial applicability. In particular, under conditions of multiple batches, multiple materials, and complex pad shapes, the existing detection system is difficult to ensure consistent recognition performance, resulting in a large amount of manual intervention in the production process, which restricts the further improvement of the intelligence and automation level of SMT production lines. Summary of the Invention

[0007] One objective of the present invention is to propose a method for detecting SMT solder joint defects based on image data. The present invention uses an adaptive region growing algorithm to accurately segment the solder joint image region, and combines it with an improved ProtoNet model to realize multi-scale feature fusion and dynamic prototype optimization, thereby effectively improving the classification accuracy and robustness of solder joint defects. By introducing adaptive feature filtering and Mahalanobis distance discrimination mechanism, the detection system's adaptability to complex pad structures and multi-batch changes is further enhanced.

[0008] According to an embodiment of the present invention, a method for detecting SMT solder joint defects based on image data includes the following steps:

[0009] S1, collect images of the SMT solder joint area and obtain solder joint image data;

[0010] S2, performing image preprocessing on the welding point image data;

[0011] S3. Using an adaptive region growing algorithm to segment the solder joint region after image preprocessing, and combining it with a boundary fitting optimization algorithm to model the solder joint region contour to generate a solder joint contour model.

[0012] S4. Based on the solder joint contour model, extract multi-level comprehensive features from the solder joint area and construct the multi-level comprehensive features into a feature vector set;

[0013] S5. Input the feature vector set into the solder joint defect detection network, use the improved ProtoNet model to predict the defect pattern of the current batch of solder joints, and use adaptive feature filtering to complete the classification and identification of solder joint defects.

[0014] Optionally, the image preprocessing includes filtering and noise reduction, contrast enhancement and normalization.

[0015] Optionally, the S3 specifically includes:

[0016] S31. In the weld image data after image preprocessing, select an initial seed point based on local texture complexity and local grayscale deviation, wherein the initial seed point satisfies:

[0017] C(x0,y0) <C th and

[0018] Among them, C(x0,y0) represents the grayscale variance in the local window at the initial seed point, C th represents the texture complexity threshold, g n (x0, y0) represents the grayscale value at the initial seed point, represents the grayscale mean, ∈ represents the grayscale deviation tolerance, and (x0, y0) represents the initial seed point;

[0019] S32, initializing a candidate point queue with the initial seed point as the starting point, adding the initial seed point to the candidate point queue, and initializing the region mask to 1;

[0020] S33. Execute adaptive region growing and dynamically adjust the grayscale difference threshold of the growing region:

[0021]

[0022] Among them, δc represents the dynamically adjusted growth grayscale difference threshold, δ0 represents the initial grayscale difference threshold, β represents the adjustment coefficient, ΔC(x c ,y c ) represents the candidate point (x c ,y c ) position, ΔC max Indicates the maximum texture change, (x c ,y c ) represents a candidate point;

[0023] For each candidate point, traverse its own neighborhood set, if it satisfies:

[0024] |g n (x n ,y n )-g n (x c ,y c )|<δ c And cos(θ c -θ n )>γ;

[0025] Among them, g n (x n ,y n ), g n (x c ,y c ) represent the grayscale values ​​of the neighborhood points and candidate points respectively, θ c ,θ n Represent the local gradient directions of candidate points and neighborhood points respectively, and γ represents the direction consistency threshold;

[0026] Then add the candidate point to the candidate point queue and set the region mask to 1, repeat until all candidate points are traversed and region growing is completed;

[0027] S34. Search for a starting point based on the region mask, where the starting point satisfies:

[0028] M(x s ,y s )=1 and

[0029] Among them, M(x s ,y s ) represents the region mask at the starting point, (x n ,y n ) indicates the starting point, A set of fields representing the starting point, means "exist", ∈ means "belong to";

[0030] S35, starting from the starting point, tracing the boundary in a clockwise direction;

[0031] During the tracking process, for each newly added boundary point, the local curvature change is calculated:

[0032]

[0033] Among them, K(x n ,y n ) represents the local curvature change, describing the three-point differential curvature at the current position, (x n-1 ,y n-1 )、(x n+1 ,y n+1 ) represent the coordinates of the previous and next boundary points respectively;

[0034] If the local curvature change is greater than the curvature change threshold, the newly added boundary points are removed;

[0035] The tracking termination conditions are returning to the starting point or the number of consecutive backtracking times exceeds the threshold or the number of boundary points reaches the maximum number of tracking steps;

[0036] S36. Calculate the fitting error and curvature change rate based on the extracted boundary point set:

[0037]

[0038] Among them, E2 represents the quadratic curve fitting error, a, b, c represent the quadratic curve fitting parameters, represents the coordinates of the i-th boundary point, N b The total number of boundary points representing the boundary point set;

[0039]

[0040] Among them, V2 represents the curvature change rate, which describes the change rate of the second-order derivative of the boundary curve. represents the second derivative of the curve, x bi+1 Represents the horizontal coordinate of the i+1th boundary point;

[0041] If the fitting error and curvature change rate are both less than the set threshold, the quadratic curve model is used for fitting:

[0042] y=ax 2 +bx+c;

[0043] Among them, (x, y) represents the coordinates of the boundary points after the quadratic curve model is fitted;

[0044] Otherwise, a cubic curve model is used for fitting:

[0045] y=a i (xx i) 3 +b i (xx i ) 2 +c i (xx i )+d i ;

[0046] Among them, a i 、b i 、c i d i Represent the parameters of the i-th segment cubic curve, x i represents the abscissa of the starting point of the i-th segment of the cubic curve;

[0047] S37. Using the parameters of the final fitting curve as a solder joint contour model.

[0048] Optionally, the multi-level comprehensive features include local geometric features, boundary curvature features, light reflection features and texture gradient features.

[0049] Optionally, the S4 specifically includes:

[0050] S41. Extract local geometric features based on the solder joint contour model M(x,y), including:

[0051] Solder point area:

[0052] A=∑ (x,y) M(x,y);

[0053] Where A represents the area of ​​the solder joint, M(x,y) represents the area mask value, and (x,y) represents the pixel coordinates;

[0054] Solder point area perimeter:

[0055]

[0056] Where P represents the perimeter of the solder joint area, represents the coordinates of the i-th boundary point, N b Indicates the number of boundary points;

[0057] Solder joint tightness:

[0058]

[0059] Among them, C f Indicates the tightness of the solder joint, A indicates the area of ​​the solder joint, and P indicates the perimeter of the solder joint;

[0060] S42. Extract boundary curvature features based on the set of boundary points of the solder joint contour, and calculate the boundary curvature for each boundary point:

[0061]

[0062] in, represents the curvature of the i-th boundary point, and Represent the coordinates of the i-1th and i+1th boundary points respectively;

[0063] S43. Extract the light reflection features based on the grayscale value and the regional mask, and calculate the regional average brightness and brightness standard deviation:

[0064]

[0065] Among them, I avg Indicates the average brightness of the solder joint area, N A represents the number of pixels in the region, Ω represents the pixel set corresponding to the region mask, g n (x,y) represents the grayscale value;

[0066]

[0067] Among them, I std Indicates the standard deviation of brightness in the solder joint area;

[0068] S44. Extract texture gradient features, calculate the grayscale gradient in the x and y directions, and calculate the regional average gradient amplitude:

[0069]

[0070] Among them, G avg represents the average gradient amplitude of the solder joint area, g x (x,y) represents the grayscale gradient in the x direction, g y (x,y) represents the grayscale gradient in the y direction;

[0071] S45. Combining the extracted local geometric features, boundary curvature features, illumination reflection features, and texture gradient features into a feature vector set in a preset order.

[0072] Optionally, the solder joint defect detection network includes a prediction module and a discrimination module.

[0073] Optionally, the prediction module uses an improved ProtoNet model to predict the defect pattern of the solder joint and generate a category prediction probability. The discrimination module performs adaptive feature filtering based on the category prediction probability, outputs the final discrimination category, and completes the classification and identification of solder joint defects.

[0074] Optionally, the solder joint defects include cold solder joints, short circuits, solder leaks, tin balls and offsets.

[0075] Optionally, the S5 specifically includes:

[0076] S51, inputting the feature vector set into the solder joint defect detection network;

[0077] S52, performing multi-scale feature transformation on each solder joint feature vector in the feature vector set;

[0078] S53. In the prediction module of the solder joint defect detection network, categories are generated based on historical solder joint defect samples. A category subset is extracted for each category, and a weighted prototype vector is calculated at each scale using a local weighted average method:

[0079]

[0080] in, represents the weighted prototype vector of category c at scale m, represents the local weighting coefficient of the jth sample, represents the solder joint feature vector of the jth sample at scale m, represents a subset of categories, represents the local weighted attenuation factor, exp represents the natural exponential function, represents the local mean eigenvector of category c at scale m, ‖·‖2 represents the Euclidean norm, and f j represents the solder joint feature vector of the jth sample;

[0081] S54. Based on the solder joint feature vectors of the current batch, calculate the local mean feature vectors by category and dynamically fine-tune the weighted prototype vectors:

[0082]

[0083] in, represents the weighted prototype vector of the adjusted category c at scale m, η represents the fine-tuning step size, Represents the local mean feature vector of the new batch category c at scale m;

[0084] S55. For each solder joint to be detected, calculate the weighted distance score between the solder joint feature vector and the dynamically adjusted weighted prototype vector at all scales:

[0085]

[0086] Among them, s i,c represents the weighted distance score between solder joint i and category c, λ m represents the fusion weight of scale m, d m represents the number of feature dimensions at scale m, represents the attention weight of the k-th feature at scale m, represents the kth solder point feature vector of solder point i at scale m, It represents the k-th prototype feature value of the adjusted category c at scale m, and M represents the total number of scales;

[0087] S56, applying a temperature-adjusted Softmax function to normalize the score-weighted distance score to obtain a class prediction probability;

[0088] S57, selecting the category corresponding to the maximum probability as the preliminary detection result;

[0089] S58. Perform adaptive feature filtering based on the category prediction probability and output the final discriminant category.

[0090] Optionally, the S58 specifically includes:

[0091] S581. Based on the category prediction probability, adaptively set the confidence threshold to filter out a high-confidence category set;

[0092] S582. Perform adaptive scale weighted fusion on the weld feature vector to obtain a fused feature vector:

[0093]

[0094] in, represents the fused feature vector, λ m (f i ) represents the scale weighting coefficient, f i represents the solder joint feature vector of the i-th sample, represents the solder joint feature vector of the jth sample at scale m, and M represents the total number of scales;

[0095] S583. Apply the attention mechanism to the fused feature vector to generate an attention weight vector, and perform element-level filtering to obtain a filtered feature vector:

[0096]

[0097] in, represents the filter feature vector, β represents the attention weight vector, and ⊙ represents the corresponding element multiplication operation;

[0098] S584. Based on the filtered feature vector, for each candidate category in the high confidence category set, calculate the Mahalanobis distance with the intra-category covariance matrix:

[0099]

[0100] Among them, d i,c represents the Mahalanobis distance with the intra-class covariance matrix, Σ c represents the intra-class covariance matrix of category c, p′ crepresents the adjusted weighted prototype vector of category c, and T represents the transposition operation;

[0101] S585. Normalize the Mahalanobis distances of each category, set a dynamic decision threshold, and output the final classification:

[0102]

[0103] in, Indicates the final discriminant category, argmin indicates the variable value when the function is maximized, represents the set of high confidence categories, c represents the category, δ i represents the dynamic decision threshold;

[0104] If no category meets the conditions, the output is "uncertain" and enters the manual review process.

[0105] The beneficial effects of the present invention are:

[0106] First, the present invention provides an SMT solder joint defect detection method based on image data. By combining an adaptive region growing algorithm with a boundary fitting optimization algorithm, the solder joint area is accurately segmented and contour modeled, fully retaining the local geometry, texture and morphological characteristics of the solder joint, significantly improving the accuracy and completeness of feature extraction, and providing a high-quality data foundation for subsequent defect detection.

[0107] Secondly, in terms of feature extraction, the present invention designs a multi-level comprehensive feature system, covering local geometric features, boundary curvature features, light reflection features and texture gradient features, which can comprehensively characterize the macroscopic appearance and microscopic local change characteristics of the solder joints, and enhance the defect detection model's perception of complex solder joint structures and minor anomalies. Through feature fusion and vectorization processing, a feature vector set with strong discriminative power is formed, further improving the detection system's ability to distinguish and identify various defect types.

[0108] In addition, in terms of defect detection network design, the present invention introduces an improved ProtoNet model, combined with multi-scale prototype modeling, dynamic prototype fine-tuning, adaptive feature weighting and Mahalanobis distance measurement mechanism, to address practical challenges such as small samples of SMT solder joint defects and large intra-category variability, and constructs a more robust and highly adaptable solder joint defect recognition framework. This method can not only achieve high-precision classification under conditions of very few labeled samples, but also adaptively adjust the detection strategy according to batch characteristics, effectively reducing the distribution drift caused by different PCB materials and different pad types, and greatly improving the generalization ability and stability of the model in actual production environments.

[0109] Finally, in the classification decision-making process, the present invention adopts an adaptive feature filtering strategy based on category prediction probability, combines the feature attention mechanism with multi-scale feature fusion, and performs refined screening and enhancement of solder joint features, so that the final classification decision can be more focused on key feature areas, thereby improving the accuracy and reliability of defect detection. By introducing the Mahalanobis distance for fine-grained category discrimination, the present invention effectively models the feature dispersion within the category, avoiding the problem of traditional Euclidean distance ignoring structural differences within the category, thereby further reducing the misjudgment rate and missed judgment rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0110] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0111] Figure 1 This is a flow chart of a SMT solder joint defect detection method based on image data proposed by the present invention;

[0112] Figure 2 A schematic diagram of the adaptive region growing and boundary modeling process of an SMT solder joint defect detection method based on image data proposed by the present invention;

[0113] Figure 3 This is a schematic diagram of the improved ProtoNet solder joint defect detection network structure of the SMT solder joint defect detection method based on image data proposed by the present invention. DETAILED DESCRIPTION

[0114] The present invention will now be described in further detail with reference to the accompanying drawings, which are simplified schematic diagrams that illustrate the basic structure of the present invention in a schematic manner.

[0115] refer to Figure 1-3 , a SMT solder joint defect detection method based on image data, comprising the following steps:

[0116] S1, collect images of the SMT solder joint area and obtain solder joint image data;

[0117] S2, performing image preprocessing on the welding point image data;

[0118] S3. Using an adaptive region growing algorithm to segment the solder joint region after image preprocessing, and combining it with a boundary fitting optimization algorithm to model the solder joint region contour to generate a solder joint contour model.

[0119] S4. Based on the solder joint contour model, extract multi-level comprehensive features from the solder joint area and construct the multi-level comprehensive features into a feature vector set;

[0120] S5. Input the feature vector set into the solder joint defect detection network, use the improved ProtoNet model to predict the defect pattern of the current batch of solder joints, and use adaptive feature filtering to complete the classification and identification of solder joint defects.

[0121] In this embodiment, the image preprocessing includes filtering and noise reduction, contrast enhancement and normalization.

[0122] In this embodiment, S3 specifically includes:

[0123] S31. In the weld image data after image preprocessing, select an initial seed point based on local texture complexity and local grayscale deviation, wherein the initial seed point satisfies:

[0124] C(x0,y0) <C th and

[0125] Among them, C(x0,y0) represents the grayscale variance in the local window at the initial seed point, C th represents the texture complexity threshold, g n (x0, y0) represents the grayscale value at the initial seed point, represents the grayscale mean, ∈ represents the grayscale deviation tolerance, and (x0, y0) represents the initial seed point;

[0126] S32, initializing a candidate point queue with the initial seed point as the starting point, adding the initial seed point to the candidate point queue, and initializing the region mask to 1;

[0127] S33. Execute adaptive region growing and dynamically adjust the grayscale difference threshold of the growing region:

[0128]

[0129] Among them, δ c represents the dynamically adjusted growth grayscale difference threshold, δ0 represents the initial grayscale difference threshold, β represents the adjustment coefficient, ΔC(x c ,y c ) represents the candidate point (x c ,y c ) position, ΔC max Indicates the maximum texture change, (x c ,y c ) represents a candidate point;

[0130] For each candidate point, traverse its own neighborhood set, if it satisfies:

[0131] |g n (x n ,y n )-gn (x c ,y c )|<δ c And cos(θ c -θ n )>γ;

[0132] Among them, g n (x n ,y n ), g n (x c ,y c ) represent the grayscale values ​​of the neighborhood points and candidate points respectively, θ c ,θ n Represent the local gradient directions of candidate points and neighborhood points respectively, and γ represents the direction consistency threshold;

[0133] Then add the candidate point to the candidate point queue and set the region mask to 1, repeat until all candidate points are traversed and region growing is completed;

[0134] S34. Search for a starting point based on the region mask, where the starting point satisfies:

[0135] M(x s ,y s )=1 and

[0136] Among them, M(x s ,y s ) represents the region mask at the starting point, (x n ,y n ) indicates the starting point, A set of fields representing the starting point, means "exist", ∈ means "belong to";

[0137] S35, starting from the starting point, tracing the boundary in a clockwise direction;

[0138] During the tracking process, for each newly added boundary point, the local curvature change is calculated:

[0139]

[0140] Among them, K(x n ,y n ) represents the local curvature change, describing the three-point differential curvature at the current position, (x n-1 ,y n-1 )、(x n+1 ,y n+1 ) represent the coordinates of the previous and next boundary points respectively;

[0141] If the local curvature change is greater than the curvature change threshold, the newly added boundary points are removed;

[0142] The tracking termination conditions are returning to the starting point or the number of consecutive backtracking times exceeds the threshold or the number of boundary points reaches the maximum number of tracking steps;

[0143] S36. Calculate the fitting error and curvature change rate based on the extracted boundary point set:

[0144]

[0145] Among them, E2 represents the quadratic curve fitting error, a, b, c represent the quadratic curve fitting parameters, represents the coordinates of the i-th boundary point, N b The total number of boundary points representing the boundary point set;

[0146]

[0147] Among them, V2 represents the curvature change rate, which describes the change rate of the second-order derivative of the boundary curve. represents the second derivative of the curve, Represents the horizontal coordinate of the i+1th boundary point;

[0148] If the fitting error and curvature change rate are both less than the set threshold, the quadratic curve model is used for fitting:

[0149] y=ax 2 +bx+c;

[0150] Among them, (x, y) represents the coordinates of the boundary points after the quadratic curve model is fitted;

[0151] Otherwise, a cubic curve model is used for fitting:

[0152] y=a i (xx i ) 3 +b i (xx i ) 2 +c i (xx i )+d i ;

[0153] Among them, a i 、b i 、c i d i Represent the parameters of the i-th segment cubic curve, x i represents the abscissa of the starting point of the i-th segment of the cubic curve;

[0154] S37. Using the parameters of the final fitting curve as a solder joint contour model.

[0155] In this embodiment, the multi-level comprehensive features include local geometric features, boundary curvature features, light reflection features and texture gradient features.

[0156] In this embodiment, the S4 specifically includes:

[0157] S41. Extract local geometric features based on the solder joint contour model M(x,y), including:

[0158] Solder point area:

[0159] A=∑ (x,y) M(x,y);

[0160] Where A represents the area of ​​the solder joint, M(x,y) represents the area mask value, and (x,y) represents the pixel coordinates;

[0161] Solder point area perimeter:

[0162]

[0163] Where P represents the perimeter of the solder joint area, represents the coordinates of the i-th boundary point, N b Indicates the number of boundary points;

[0164] Solder joint tightness:

[0165]

[0166] Among them, C f Indicates the tightness of the solder joint, A indicates the area of ​​the solder joint, and P indicates the perimeter of the solder joint;

[0167] S42. Extract boundary curvature features based on the set of boundary points of the solder joint contour, and calculate the boundary curvature for each boundary point:

[0168]

[0169] in, represents the curvature of the i-th boundary point, and Represent the coordinates of the i-1th and i+1th boundary points respectively;

[0170] S43. Extract the light reflection features based on the grayscale value and the regional mask, and calculate the regional average brightness and brightness standard deviation:

[0171]

[0172] Among them, I avg Indicates the average brightness of the solder joint area, N A represents the number of pixels in the region, Ω represents the pixel set corresponding to the region mask, gn (x,y) represents the grayscale value;

[0173]

[0174] Among them, I std Indicates the standard deviation of brightness in the solder joint area;

[0175] S44. Extract texture gradient features, calculate the grayscale gradient in the x and y directions, and calculate the regional average gradient amplitude:

[0176]

[0177] Among them, G avg represents the average gradient amplitude of the solder joint area, g x (x,y) represents the grayscale gradient in the x direction, g y (x,y) represents the grayscale gradient in the y direction;

[0178] S45. Combining the extracted local geometric features, boundary curvature features, illumination reflection features, and texture gradient features into a feature vector set in a preset order.

[0179] In this embodiment, the solder joint defect detection network includes a prediction module and a discrimination module.

[0180] In this embodiment, the prediction module uses an improved ProtoNet model to predict the defect pattern of the solder joint and generate a category prediction probability. The discrimination module performs adaptive feature filtering based on the category prediction probability, outputs the final discrimination category, and completes the classification and identification of solder joint defects.

[0181] In this embodiment, the solder joint defects include cold solder joints, short circuits, solder leaks, tin balls, and offsets.

[0182] In this embodiment, the S5 specifically includes:

[0183] S51, inputting the feature vector set into the solder joint defect detection network;

[0184] S52, performing multi-scale feature transformation on each solder joint feature vector in the feature vector set;

[0185] S53. In the prediction module of the solder joint defect detection network, categories are generated based on historical solder joint defect samples. A category subset is extracted for each category, and a weighted prototype vector is calculated at each scale using a local weighted average method:

[0186]

[0187] in, represents the weighted prototype vector of category c at scale m, represents the local weighting coefficient of the jth sample, represents the solder joint feature vector of the jth sample at scale m, represents a subset of categories, represents the local weighted attenuation factor, exp represents the natural exponential function, represents the local mean eigenvector of category c at scale m, ‖·‖2 represents the Euclidean norm, and f j represents the solder joint feature vector of the jth sample;

[0188] S54. Based on the solder joint feature vectors of the current batch, calculate the local mean feature vectors by category and dynamically fine-tune the weighted prototype vectors:

[0189]

[0190] in, represents the weighted prototype vector of the adjusted category c at scale m, η represents the fine-tuning step size, Represents the local mean feature vector of the new batch category c at scale m;

[0191] S55. For each solder joint to be detected, calculate the weighted distance score between the solder joint feature vector and the dynamically adjusted weighted prototype vector at all scales:

[0192]

[0193] Among them, s i,c represents the weighted distance score between solder joint i and category c, λ m represents the fusion weight of scale m, d m represents the number of feature dimensions at scale m, represents the attention weight of the k-th feature at scale m, represents the kth solder point feature vector of solder point i at scale m, It represents the k-th prototype feature value of the adjusted category c at scale m, and M represents the total number of scales;

[0194] S56, applying a temperature-adjusted Softmax function to normalize the score-weighted distance score to obtain a class prediction probability;

[0195] S57, selecting the category corresponding to the maximum probability as the preliminary detection result;

[0196] S58. Perform adaptive feature filtering based on the category prediction probability and output the final discriminant category.

[0197] In this embodiment, the S58 specifically includes:

[0198] S581. Based on the category prediction probability, adaptively set the confidence threshold to filter out a high-confidence category set;

[0199] S582. Perform adaptive scale weighted fusion on the weld feature vector to obtain a fused feature vector:

[0200]

[0201] in, represents the fused feature vector, λ m (f i ) represents the scale weighting coefficient, f i represents the solder joint feature vector of the i-th sample, represents the solder joint feature vector of the jth sample at scale m, and M represents the total number of scales;

[0202] S583. Apply the attention mechanism to the fused feature vector to generate an attention weight vector, and perform element-level filtering to obtain a filtered feature vector:

[0203]

[0204] in, represents the filter feature vector, β represents the attention weight vector, and ⊙ represents the corresponding element multiplication operation;

[0205] S584. Based on the filtered feature vector, for each candidate category in the high confidence category set, calculate the Mahalanobis distance with the intra-category covariance matrix:

[0206]

[0207] Among them, d i,c represents the Mahalanobis distance with the intra-class covariance matrix, Σ c represents the intra-class covariance matrix of category c, p′ c represents the adjusted weighted prototype vector of category c, and T represents the transposition operation;

[0208] S585. Normalize the Mahalanobis distances of each category, set a dynamic decision threshold, and output the final classification:

[0209]

[0210] in, Indicates the final discriminant category, argmin indicates the variable value when the function is maximized, represents the set of high confidence categories, c represents the category, δ i represents the dynamic decision threshold;

[0211] If no category meets the conditions, the output is "uncertain" and enters the manual review process.

[0212] Example 1:

[0213] In order to verify the feasibility of the present invention in implementation, the present invention was applied to the solder joint detection line of the SMT workshop of a well-known electronics manufacturer from September 2024 to December 2024 for actual application test. The test object was a batch-produced double-sided mounting high-density PCB board. The main board type was FR4 epoxy glass fiber board with a thickness of 1.6 mm. The pad types included OSP process pads, gold-plated pads and HASL (hot air leveling) pads. The solder joint types covered 0402, 0603 standard chip packages and some BGA packages.

[0214] During the implementation of the present invention, an industrial camera is first used to capture images of the solder joint area of ​​the PCB board. The image resolution is set to 2448×2048 pixels, the exposure time is optimized to 2ms to adapt to the lighting conditions of the workshop, and a mixed arrangement of coaxial light source and annular light source is used. The collected solder joint image data is processed by the image preprocessing method proposed in the present invention, including noise reduction based on adaptive filtering, local contrast enhancement operation and normalization processing, so that the brightness and texture details of the solder joint images under different pad materials remain stable and consistent.

[0215] An adaptive region growing algorithm is then applied to the preprocessed images for solder joint segmentation. For each image, the seed point with the largest local grayscale variance and smallest grayscale deviation is automatically selected. The grayscale difference threshold is dynamically adjusted during the region growing process, and combined with the consistency of neighborhood gradient directions, this effectively avoids the overgrowth and mis-expansion problems caused by pad reflections and contamination, as is the case with traditional methods. The segmented solder joint boundaries are extracted using clockwise boundary tracing, and fitting accuracy is assessed based on local curvature changes and quadratic curve fitting errors. For complex boundary regions, a cubic curve fitting strategy is automatically used, enabling adaptive adjustment of contour modeling.

[0216] After the solder joint contour modeling is completed, multi-level comprehensive features are further extracted based on the extracted boundary point set, including the local geometric features of the solder joint (area, perimeter, compactness), boundary curvature features (local curvature change curve), light reflection features (regional average brightness and brightness standard deviation) and texture gradient features (directional grayscale changes). These features are combined into a feature vector set according to a preset structure to provide a high-dimensional description for subsequent defect identification.

[0217] During the detection phase, the feature vector set is input into the improved ProtoNet solder joint defect detection network. First, in the prediction module, based on the defect samples annotated in historical production data (a total of 3,000 image samples, including five types of defects: cold solder joints, short circuits, leaking solder joints, solder balls, and offset), a local weighted prototype is generated for each defect type in the multi-scale feature space. Based on the characteristics of the solder joints in this batch, the prototype position is adjusted through a dynamic fine-tuning strategy to adapt to the distribution drift between batches. During the inference process, a weighted distance matching method with feature attention is used to integrate feature information at multiple scales. The predicted probability of each category is output after temperature Softmax normalization.

[0218] During the initial inspection phase, the maximum predicted probability is used to determine the preliminary classification result, which then enters the discrimination module. Within this module, the confidence level is adaptively set based on the predicted probability to filter the set, retaining high-confidence categories. Adaptive scale fusion is performed on solder joint features, and an attention mechanism is introduced for feature filtering. Finally, the Mahalanobis distance is used to calculate the subtle differences between the filtered features and the dynamic prototypes of each candidate category. The normalized Mahalanobis distance score and the dynamic decision threshold are combined to output the final solder joint defect category, automatically triggering manual review in abnormal situations.

[0219] Table 1 Comparison of experimental data between the present invention and the traditional method

[0220] Comparison Project Traditional CNN detection method Detection method of the present invention Detection accuracy (%) 96.95 99.45 False detection rate (%) 2.20 0.28 Missed detection rate (%) 0.85 0.27 Average detection time per board (seconds) 2.8 2.3 Board adaptability (OSP board accuracy %) 97.10 99.38 Plate adaptability (accuracy of gold-plated plate%) 97.35 99.49 Plate adaptability (HASL plate accuracy %) 97.20 99.52 Small sample new defect adaptation time (hours) 8 2 Adaptability to process variation medium high

[0221] From the perspective of overall detection accuracy, the solder joint defect detection method based on the improved ProtoNet model of the present invention achieved a high accuracy of 99.45% in standard batch tests, which is significantly better than the 96.95% level of the traditional CNN detection method. The accuracy rate was improved by nearly 2.5 percentage points, fully demonstrating the comprehensive performance advantages of the present invention in feature extraction and classification recognition.

[0222] In terms of false detection rate and missed detection rate, the false detection rate of the detection system of the present invention is reduced to 0.28%, and the missed detection rate is only 0.27%, which are reduced by 1.92% and 0.58% respectively compared with the traditional CNN method, effectively reducing the production interference and quality risks caused by false or missed detection, and further ensuring the product consistency and reliability in the electronic manufacturing process.

[0223] In terms of inspection speed, this invention reduces the average inspection time per PCB to 2.3 seconds by optimizing feature vector processing and improving the classification decision process, a reduction of approximately 18% compared to the 2.8 seconds of the traditional method. This increased inspection efficiency not only helps meet the real-time inspection requirements of high-speed production lines, but also provides strong support for the intelligent layout of subsequent production lines.

[0224] Regarding adaptability to different board types, the present invention achieved detection accuracy rates of 99.38%, 99.49%, and 99.52% on OSP boards, gold-plated boards, and HASL boards, respectively. These rates represent improvements of approximately 2% or more compared to traditional CNN methods, with minimal differences in accuracy. This demonstrates the present invention's high stability and good generalization capabilities when dealing with variations in pad process materials. Regarding its adaptability to new defects in small sample sizes, the present invention introduces a dynamic prototype fine-tuning mechanism, shortening the adaptation time for new defects from the traditional method's 8 hours to 2 hours. This significantly reduces manual intervention costs and system update cycles, and enhances the detection system's online self-evolution capabilities.

[0225] In terms of adaptability to process variations, the present invention can dynamically adjust the detection strategy based on changes in the local characteristics of the solder joint image. Compared to traditional CNN methods that can only adapt to batch fluctuations to a moderate degree, the present invention exhibits a higher tolerance for process variations, improving the application effect of the detection system in a multi-process, multi-batch environment. In summary, through comprehensive experimental data comparison, the present invention has demonstrated significant superiority in detection accuracy, detection speed, adaptability, stability, and small sample adaptation capabilities. It can significantly improve the level of automated SMT solder joint detection and help the electronics manufacturing industry achieve more efficient and intelligent quality control goals.

[0226] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with the technical field, within the technical scope disclosed by the present invention, who makes equivalent replacements or changes based on the technical solution and inventive concept of the present invention, should be covered by the scope of protection of the present invention.

Claims

1. A method for detecting SMT solder joint defects based on image data, characterized in that: The steps include: S1, collect images of the SMT solder joint area and obtain solder joint image data; S2, performing image preprocessing on the welding point image data; S3. Using an adaptive region growing algorithm to segment the solder joint region after image preprocessing, and combining it with a boundary fitting optimization algorithm to model the solder joint region contour to generate a solder joint contour model. S4. Based on the solder joint contour model, extract multi-level comprehensive features from the solder joint area and construct the multi-level comprehensive features into a feature vector set; S5. Input the feature vector set into the solder joint defect detection network, use the improved ProtoNet model to predict the defect pattern of the current batch of solder joints, and use adaptive feature filtering to complete the classification and identification of solder joint defects.

2. The SMT solder joint defect detection method based on image data according to claim 1, characterized in that: The image preprocessing includes filtering, noise reduction, contrast enhancement and normalization.

3. The SMT solder joint defect detection method based on image data according to claim 1, characterized in that: The S3 specifically includes: S31. Selecting an initial seed point in the weld image data after image preprocessing based on local texture complexity and local grayscale deviation; S32, initializing a candidate point queue with the initial seed point as the starting point, adding the initial seed point to the candidate point queue, and initializing the region mask to 1; S33. Execute adaptive region growing and dynamically adjust the grayscale difference threshold of the growing region: Among them, δ c represents the dynamically adjusted growth grayscale difference threshold, δ0 represents the initial grayscale difference threshold, β represents the adjustment coefficient, ΔC(x c ,y c ) represents the candidate point (x c ,y c ) position, ΔC max Indicates the maximum texture change, (x c ,y c ) represents a candidate point; For each candidate point, traverse its own neighborhood set, if it satisfies: |g n (x n ,y n )-g n (x c ,y c )|<δ c And cos(θ c -θ n )>c; Among them, g n (x n ,y n ), g n (x c ,y c ) represent the grayscale values ​​of the neighborhood points and candidate points respectively, θ c ,θ n Represent the local gradient directions of candidate points and neighborhood points respectively, and γ represents the direction consistency threshold; Then add the candidate point to the candidate point queue and set the region mask to 1, repeat until all candidate points are traversed and region growing is completed; S34, searching for a starting point based on the regional mask; S35, starting from the starting point, tracing the boundary in a clockwise direction; During the tracking process, for each newly added boundary point, the local curvature change is calculated; If the local curvature change is greater than the curvature change threshold, the newly added boundary points are removed; The tracking termination conditions are returning to the starting point or the number of consecutive backtracking times exceeds the threshold or the number of boundary points reaches the maximum number of tracking steps; S36. Calculate the fitting error and curvature change rate respectively according to the extracted boundary point set; If the fitting error and curvature change rate are both less than the set threshold, the quadratic curve model is used for fitting; Otherwise, a cubic curve model is used for fitting; S37. Using the parameters of the final fitting curve as a solder joint contour model.

4. The SMT solder joint defect detection method based on image data according to claim 1, characterized in that: The multi-level comprehensive features include local geometric features, boundary curvature features, light reflection features and texture gradient features.

5. The SMT solder joint defect detection method based on image data according to claim 1, characterized in that: The S4 specifically includes: S41. Extract local geometric features based on the solder joint contour model; S42, extracting boundary curvature features based on the set of boundary points of the weld point contour, and calculating the boundary curvature for each boundary point; S43, extracting light reflection features based on the grayscale value and the regional mask, and calculating the regional average brightness and brightness standard deviation; S44, extracting texture gradient features, calculating grayscale gradients in the x-direction and y-direction, and calculating the regional average gradient amplitude; S45. Combining the extracted local geometric features, boundary curvature features, illumination reflection features, and texture gradient features into a feature vector set in a preset order.

6. The SMT solder joint defect detection method based on image data according to claim 1, characterized in that: The solder joint defect detection network includes a prediction module and a discrimination module.

7. The SMT solder joint defect detection method based on image data according to claim 6, characterized in that: The prediction module uses an improved ProtoNet model to predict the defect pattern of the solder joint and generate category prediction probabilities. The discrimination module performs adaptive feature filtering based on the category prediction probabilities and outputs the final discrimination category to complete the classification and identification of solder joint defects.

8. The SMT solder joint defect detection method based on image data according to claim 7, characterized in that: The solder joint defects include cold solder joints, short circuits, solder leaks, tin balls and offset.

9. The SMT solder joint defect detection method based on image data according to claim 1, characterized in that: The S5 specifically includes: S51, inputting the feature vector set into the solder joint defect detection network; S52, performing multi-scale feature transformation on each solder joint feature vector in the feature vector set; S53. In the prediction module of the solder joint defect detection network, categories are generated based on historical solder joint defect samples. A category subset is extracted for each category, and a weighted prototype vector is calculated at each scale using a local weighted average method: in, represents the weighted prototype vector of category c at scale m, represents the local weighting coefficient of the jth sample, represents the solder joint feature vector of the jth sample at scale m, represents a subset of categories, represents the local weighted attenuation factor, exp represents the natural exponential function, represents the local mean eigenvector of category c at scale m, ‖·‖2 represents the Euclidean norm, and f j represents the solder joint feature vector of the jth sample; S54. Based on the solder joint feature vectors of the current batch, calculate the local mean feature vectors by category and dynamically fine-tune the weighted prototype vectors: in, represents the weighted prototype vector of the adjusted category c at scale m, η represents the fine-tuning step size, Represents the local mean feature vector of the new batch category c at scale m; S55. For each solder joint to be detected, calculate the weighted distance score between the solder joint feature vector and the dynamically adjusted weighted prototype vector at all scales: Among them, s i,c represents the weighted distance score between solder joint i and category c, λ m represents the fusion weight of scale m, d m represents the number of feature dimensions at scale m, represents the attention weight of the k-th feature at scale m, represents the kth solder point feature vector of solder point i at scale m, It represents the k-th prototype feature value of the adjusted category c at scale m, and M represents the total number of scales; S56, applying a temperature-adjusted Softmax function to normalize the score-weighted distance score to obtain a class prediction probability; S57, selecting the category corresponding to the maximum probability as the preliminary detection result; S58. Perform adaptive feature filtering based on the category prediction probability and output the final discriminant category.

10. The SMT solder joint defect detection method based on image data according to claim 9, characterized in that: The S58 specifically includes: S581. Based on the category prediction probability, adaptively set the confidence threshold to filter out a high-confidence category set; S582, performing adaptive scale weighted fusion on the weld point feature vector to obtain a fused feature vector; S583, applying an attention mechanism to the fused feature vector to generate an attention weight vector, and performing element-level filtering to obtain a filtered feature vector; S584. Based on the filtered feature vector, for each candidate category in the high confidence category set, calculate the Mahalanobis distance with the intra-category covariance matrix: Among them, d i,c represents the Mahalanobis distance with the intra-class covariance matrix, Σ c represents the intra-class covariance matrix of category c, p′ c represents the adjusted weighted prototype vector of category c, and T represents the transposition operation; S585, normalizing the Mahalanobis distances of each category, setting a dynamic decision threshold, and outputting the final discrimination category; If no category meets the conditions, the output is "uncertain" and the manual review process begins.

Images