PCBA solder joint defect detection method and system based on image recognition

Abstract

The application relates to the technical field of electronic manufacturing detection, in particular to a PCBA solder joint defect detection method and system based on image recognition, which comprises the following steps: obtaining a high-resolution surface image of a PCBA board to be detected, and positioning and segmenting independent image areas of each solder joint; performing multi-scale feature extraction on each area, fusing macroscopic morphological features and microscopic texture features to generate a comprehensive feature descriptor set; inputting a pre-trained deep classification network to obtain an initial defect classification result and a confidence score; constructing a cross-solder joint space correlation model to analyze adjacent area defect correlation features, optimizing the initial result to generate a final classification; and positioning the defect solder joint and labeling the type and coordinates on the original image. Through multi-scale feature fusion and space correlation optimization, the application improves the consistency of complex defect expression and classification, and realizes accurate detection.

Description

Technical Field

[0001] This application relates to the field of electronic manufacturing inspection technology, and in particular to a method and system for detecting PCBA solder joint defects based on image recognition. Background Technology

[0002] As a core component of electronic devices, the quality of the solder joints on PCBA boards directly affects product reliability, requiring efficient defect detection through image recognition technology.

[0003] Existing image recognition methods mostly achieve detection by acquiring surface images, segmenting solder joint regions, extracting features, and classifying them. However, these methods have significant limitations in practical applications. Conventional feature extraction often relies on single-scale or single-type features, making it difficult to simultaneously capture macroscopic structural anomalies and microscopic detail defects. This results in one-sided feature representation and insufficient ability to distinguish complex defects. Furthermore, existing methods often classify individual solder joints independently, without considering the spatial relationships between adjacent solder joints. This makes them prone to isolated misclassifications due to local noise or changes in illumination, leading to a lack of spatial consistency in the classification results. Summary of the Invention

[0004] The purpose of this application is to address the shortcomings of existing technologies by proposing a PCBA solder joint defect detection method and system based on image recognition.

[0005] To achieve the above objectives, this application adopts the following technical solution: a PCBA solder joint defect detection method based on image recognition, comprising: acquiring a high-resolution surface image of the PCBA board to be inspected, wherein the high-resolution surface image contains complete optical information of multiple solder joint regions; performing solder joint region localization processing on the high-resolution surface image, identifying and segmenting the independent image region corresponding to each solder joint in the high-resolution surface image, and generating a set of solder joint region images; performing multi-scale feature extraction processing on each independent image region in the set of solder joint region images to obtain a comprehensive feature descriptor set containing macroscopic morphological features and microscopic texture features; inputting the comprehensive feature descriptor set into a pre-trained deep classification network for defect pattern matching analysis, generating an initial defect classification result and confidence score for each solder joint region; constructing a cross-solder joint spatial association model based on the initial defect classification result and confidence score, analyzing the defect association features between adjacent solder joint regions, optimizing the spatial consistency of the initial defect classification result, and generating an optimized defect classification result; locating the solder joint region with defects according to the optimized defect classification result, and generating annotation information containing defect type and location coordinates on the original high-resolution surface image.

[0006] As a further aspect of this application, the step of performing solder joint region localization processing on the high-resolution surface image, identifying and segmenting the independent image region corresponding to each solder joint in the high-resolution surface image, and generating a set of solder joint region images includes: processing the high-resolution surface image using an edge detection algorithm to extract candidate edge information of all solder joint contours in the image; performing shape matching and filtering on the candidate edge information based on prior knowledge of the geometric shape of the solder joint contours to obtain accurate closed curves of the solder joint contours; cropping the pixel region corresponding to each solder joint from the high-resolution surface image using image masking technology according to the closed curves of the solder joint contours to form the independent image region; and performing size normalization processing on all independent image regions to ensure that each solder joint region image has the same pixel size, thereby forming the set of solder joint region images.

[0007] As a further aspect of this application, the step of performing multi-scale feature extraction processing on each independent image region in the solder joint region image set to obtain a comprehensive feature descriptor set containing macroscopic morphological features and microscopic texture features includes: applying Gaussian filters of different sizes to each independent image region for multi-scale blurring processing to generate blurred image pyramids corresponding to different scale spaces; extracting geometric morphological features of the solder joint region at each layer of the blurred image pyramid, the geometric morphological features including area, perimeter, roundness, and aspect ratio of the minimum bounding rectangle; simultaneously applying a local binary mode algorithm to each independent image region to extract microscopic texture features of the solder joint surface, the microscopic texture features including texture contrast, uniformity, and directionality; serially concatenating and standardizing the geometric morphological features and microscopic texture features extracted at different scales for the same solder joint region to generate the comprehensive feature descriptor of the solder joint region, and the comprehensive feature descriptors of all solder joint regions constitute the comprehensive feature descriptor set.

[0008] As a further aspect of this application, the step of inputting the comprehensive feature descriptor set into a pre-trained deep classification network for defect pattern matching analysis to generate an initial defect classification result and confidence score for each solder joint region includes: the pre-trained deep classification network comprising multiple fully connected layers and a Softmax output layer; the comprehensive feature descriptor set is used as the input to the deep classification network, and is sequentially passed through the multiple fully connected layers for nonlinear transformation and feature dimensionality reduction; in the Softmax output layer, the dimensionality-reduced feature vectors are mapped to probability distributions belonging to different defect types, including cold solder joints, solder bridging, insufficient solder, excessive solder, and normal solder joints; the defect type with the highest probability is used as the initial defect classification result, and the corresponding probability value is used as the confidence score.

[0009] As a further aspect of this application, the step of constructing a cross-solder joint spatial association model based on the initial defect classification result and confidence score, analyzing the defect association characteristics between adjacent solder joint regions, optimizing the spatial consistency of the initial defect classification result, and generating an optimized defect classification result includes: constructing a solder joint adjacency graph based on the physical location coordinates of the solder joints in the high-resolution surface image, where nodes represent solder joints and edges represent the adjacency relationships between solder joints; assigning the initial defect classification result and the confidence score as an initial state to each node in the solder joint adjacency graph; defining a defect propagation energy function, which calculates the inconsistency cost between the current node state and the adjacent node states based on the assumption that adjacent solder joints have similar defect types; minimizing the defect propagation energy function using a graph optimization algorithm, iteratively updating the defect classification state of each node until convergence; and using the converged defect classification state of each node as the optimized defect classification result.

[0010] As a further aspect of this application, the definition of the defect propagation energy function is based on the assumption that adjacent solder joints have similar defect types. The function calculates the inconsistency cost between the current node state and the states of adjacent nodes, including: for any edge in the solder joint adjacency graph, defining an inter-node inconsistency term, which is proportional to the degree of difference in defect type labels corresponding to two adjacent nodes; for any node in the solder joint adjacency graph, defining a node data fidelity term, which is inversely proportional to the difference between the corresponding node's defect type label and its initial defect classification result, and weighted by the confidence score of the corresponding node; the defect propagation energy function is the sum of the node data fidelity terms of all nodes and the inter-node inconsistency terms of all edges.

[0011] As a further aspect of this application, the step of locating defective solder joint regions based on the optimized defect classification results and generating annotation information containing defect type and location coordinates on the original high-resolution surface image includes: comparing the optimized defect classification results with preset normal solder joint type labels, and filtering out all solder joint regions whose defect type is not a normal solder joint; determining the location bounding box of each defective solder joint region in the image based on the coordinate position of each independent image region recorded during the solder joint region location processing in the original high-resolution surface image; generating an annotation entry for each defective solder joint region, the annotation entry containing a text description of the defect type of the solder joint region and the coordinate data of its location bounding box; and superimposing all generated annotation entries onto the original high-resolution surface image to form a visual defect detection report image.

[0012] As a further aspect of this application, after acquiring the high-resolution surface image of the PCBA board to be inspected, the method further includes: preprocessing the high-resolution surface image, wherein the preprocessing includes illumination equalization processing and background noise suppression processing; the illumination equalization processing calculates the global and local histograms of the image and applies histogram specification technology to make the overall illumination distribution of the image uniform; the background noise suppression processing adopts a nonlocal mean denoising algorithm to smooth the noise in the background area of ​​the image while preserving the details of the solder joint edges.

[0013] As a further aspect of this application, the training process of the pre-trained deep classification network includes: collecting image samples containing various PCBA solder joint defect types to construct a training dataset, wherein each sample in the training dataset contains a solder joint region image and its corresponding true defect type label; using the training dataset to iteratively train the initialized deep classification network, and in each iteration, calculating the cross-entropy loss between the defect type predicted by the network and the true defect type label; using a gradient descent algorithm to update the weight parameters of the deep classification network to minimize the cross-entropy loss, until the classification accuracy of the network on the validation dataset converges to a preset threshold, thereby obtaining the pre-trained deep classification network.

[0014] As a further aspect of this application, this application also includes a PCBA solder joint defect detection system based on image recognition. The system includes a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the computer program, it implements the steps of the PCBA solder joint defect detection method based on image recognition described above.

[0015] Compared with the prior art, the advantages and positive effects of this application are as follows.

[0016] Multi-scale feature extraction is performed on each independent image region in the solder joint image set, simultaneously capturing macroscopic morphological features and microscopic texture features, and integrating them into a comprehensive feature descriptor set. Macroscopic morphological features encompass global geometric attributes such as the overall outline, size parameters, and height symmetry of the solder joint, while microscopic texture features include the distribution of solder particles on the surface, gloss uniformity, and local texture details of microcracks or pores. This multi-scale fusion enables the feature descriptors to simultaneously reflect structural anomalies and surface detail defects. Compared to single feature extraction, it can more comprehensively characterize the essence of complex defects, enhance the ability to express hidden and complex defects, and provide richer discriminative criteria for subsequent classification.

[0017] Based on the initial defect classification results and confidence scores, a cross-solder joint spatial correlation model is constructed to analyze the defect correlation characteristics between adjacent solder joint regions, and to optimize the spatial consistency of the initial classification results. This model identifies spatial correlations such as the continuous distribution patterns of similar defects in adjacent solder joints and chain defect patterns consistent with circuit layouts, adjusting isolated high-confidence misjudgments or strengthening low-confidence judgments that conform to correlation rules. This process extends single-point classification to spatial context collaborative judgment, reducing false detections caused by local interference, making the final defect classification results more consistent with the actual defect distribution logic of solder joint clusters, and improving the overall consistency and reliability of the detection results.

[0018] This application employs multi-scale feature extraction processing to simultaneously capture macroscopic morphological features and microscopic texture features of solder joints and integrate them into a comprehensive feature descriptor, thus solving the problem of incomplete feature representation. By constructing a cross-solder joint spatial correlation model, it analyzes the defect correlation features of adjacent solder joints to optimize the initial classification results, solving the problem of neglecting spatial consistency in single-point independent classification, thereby improving the accuracy and robustness of defect detection. Attached Figure Description

[0019] Figure 1 This is a flowchart of the PCBA solder joint defect detection method based on image recognition described in this application;

[0020] Figure 2 A flowchart for multi-scale feature extraction processing;

[0021] Figure 3 The graph shows the loss variation of the PCBA solder joint defect detection model during the training process.

[0022] Figure 4 Scatter plot of node data fidelity items during the construction phase of the cross-weld joint spatial association model;

[0023] Figure 5 A comparison chart showing the defect identification accuracy before and after optimization of the spatial correlation model in the PCBA solder joint defect detection process. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0025] In the description of this application, it should be understood that the terms "length," "width," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicating orientation or positional relationships based on the orientation or positional relationships shown in the accompanying drawings, are used only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this application. Furthermore, in the description of this application, "a plurality of" means two or more, unless otherwise explicitly specified.

[0026] See Figure 1 The process involves acquiring a high-resolution surface image of the PCBA board to be inspected, containing complete optical information of multiple solder joint regions. Next, solder joint region localization processing is performed on the high-resolution surface image, identifying and segmenting the independent image region corresponding to each solder joint, thus generating a set of solder joint region images. Then, multi-scale feature extraction processing is performed on each independent image region in the solder joint region image set to obtain a comprehensive feature descriptor set containing macroscopic morphological features and microscopic texture features. Subsequently, the comprehensive feature descriptor set is input into a pre-trained deep classification network for defect pattern matching analysis, generating an initial defect classification result and corresponding confidence score for each solder joint region. Based on the initial defect classification results and confidence scores, a cross-solder joint spatial association model is constructed to analyze the defect association features between adjacent solder joint regions, and spatial consistency optimization is performed on the initial defect classification results to generate optimized defect classification results. Finally, based on the optimized defect classification results, the solder joint regions with defects are located, and annotation information containing defect type and location coordinates is generated on the original high-resolution surface image.

[0027] In one embodiment of this application, after acquiring a high-resolution surface image of the PCBA board to be inspected, the high-resolution surface image is preprocessed. The preprocessing includes illumination equalization and background noise suppression. Illumination equalization calculates the global and local histograms of the high-resolution surface image and applies histogram specification techniques to ensure uniform illumination distribution across the entire high-resolution surface image. Background noise suppression employs a non-local means denoising algorithm to smooth noise in the background area of ​​the high-resolution surface image while preserving solder joint edge details. The non-local means denoising algorithm calculates a denoised value for each pixel in the high-resolution surface image. The denoised value is based on a weighted average of neighboring pixels, with the weights determined by the similarity between pixel blocks. The weights are calculated using the following formula: .in, This represents the weight between pixel i and pixel j. This represents the pixel block vector centered at pixel i. This represents the pixel block vector centered at pixel j. This indicates that the filtering parameters control the rate of weight decay. It can be understood that the weight calculation formula ensures that similar pixel blocks contribute more to the noise reduction of the current pixel.

[0028] In some embodiments, the specific steps of illumination equalization processing include calculating a global histogram of the high-resolution surface image to obtain the brightness distribution of the entire image, dividing the high-resolution surface image into multiple local regions and calculating a local histogram for each local region, and matching the histogram of each local region to a standardized reference histogram through histogram specification techniques to adjust the brightness values ​​of the local regions so that the illumination conditions of the high-resolution surface image tend to be consistent. The implementation of the nonlocal mean denoising algorithm in background noise suppression processing relies on a similarity weight function, which measures the degree of difference between two pixel blocks. Optionally, the similarity weight function can be defined based on Gaussian weighted Euclidean distance to enhance the algorithm's robustness to noise. In a specific implementation, the preprocessed high-resolution surface image undergoes solder joint region localization processing, and an edge detection algorithm is used to process the high-resolution surface image, extracting candidate edge information of all solder joint contours in the high-resolution surface image. The edge detection algorithm can be the Canny operator, Sobel operator, or Prewitt operator, which identifies edge points by calculating the gradient magnitude and direction of the high-resolution surface image. Based on the prior knowledge of the geometric shape of the solder joint profile, the candidate edge information is shape matched and filtered to obtain an accurate closed curve of the solder joint profile. The prior knowledge of the geometric shape of the solder joint profile includes that the solder joint usually presents an approximate circle or ellipse with a specific area and perimeter range. The shape matching and filtering process filters out edges that do not meet the conditions by comparing the geometric properties of the candidate edges with the prior knowledge and connects the edges that meet the conditions into a closed curve.

[0029] In some embodiments, shape matching and filtering use template matching or Hough transform to identify solder joint contours. For example, for circular solder joints, Hough circle transform can be used to detect circular contours in candidate edge information. Based on the closed curve of the solder joint contour, image masking techniques are used to crop the pixel region corresponding to each solder joint from the high-resolution surface image, forming an independent image region. The image masking technique creates a binary mask corresponding to the closed curve of the solder joint contour, where the value inside the mask is 1 and the value outside the mask is 0. The mask is then applied to the high-resolution surface image to extract the pixel values ​​of the region inside the mask. Size normalization is performed on all independent image regions to ensure that each solder joint region image has the same pixel size, forming a set of solder joint region images. Size normalization uses interpolation algorithms such as bilinear interpolation or nearest neighbor interpolation to adjust each independent image region to a preset fixed size. Optionally, the parameter settings in the edge detection algorithm can be adjusted according to the resolution and noise level of the high-resolution surface image to optimize the edge extraction effect. It is understood that the prior knowledge of the geometric shape of the solder joint contour can be defined based on the solder joint design specifications in actual production, such as the standard diameter and shape tolerance of the solder joint.

[0030] See Figure 2 In one embodiment of this application, multi-scale feature extraction processing is performed on each independent image region in the solder joint region image set. The processing begins by applying Gaussian filters of different sizes to each independent image region for multi-scale blurring. The kernel size of the Gaussian filter can be set to multiple different values, corresponding to the generation of blurred images at multiple scales, thus forming a blurred image pyramid. The bottom layer of the blurred image pyramid is the original independent image region, while the upper layer is the progressively blurred image. For example, for an independent image region with a size of 64 pixels × 64 pixels, the standard deviation of the Gaussian filter can be 1.0 pixel, 2.0 pixel, and 4.0 pixels respectively to generate a blurred image pyramid at three scales. In some embodiments, the kernel size of the Gaussian filter is selected based on the size of the independent image region and the typical feature size of the solder joint. The number and specific values ​​of the kernel size can be adjusted according to actual detection needs. Multi-scale blurring processing can capture the morphological features of the solder joint region at different resolutions. At each level of the blurred image pyramid, geometric features of the solder joint region are extracted. These features include area, perimeter, roundness, and the aspect ratio of the minimum bounding rectangle. The area is obtained by calculating the number of pixels in the binary mask of the solder joint region. The perimeter is obtained by calculating the pixel boundary length of the solder joint outline. The roundness is calculated by multiplying four times pi by the area and then dividing by the square of the perimeter. The aspect ratio of the minimum bounding rectangle is obtained by fitting the minimum bounding rectangle of the solder joint region and calculating its width-to-height ratio. In essence, these geometric features describe the overall shape and size attributes of the solder joint at different scales.

[0031] In specific implementations, a Local Binary Pattern (LCP) algorithm is applied to extract the microscopic texture features of the solder joint surface on each independent image region. The LCP algorithm generates a local binary pattern code by comparing the grayscale values ​​of each pixel with its neighboring pixels. The microscopic texture features include texture contrast, uniformity, and directionality. Texture contrast is measured by calculating the standard deviation of local grayscale differences; uniformity is measured by statistically analyzing the proportion of uniform patterns in the LCP code; and directionality is measured by analyzing the main direction of the local gradient direction histogram. Optionally, the LCP algorithm can use circular neighborhoods instead of square neighborhoods to enhance rotation invariance. The neighborhood radius and the number of sampling points can be configured to balance feature discrimination and computational efficiency. In some embodiments, the calculation of texture contrast, uniformity, and directionality can be based on the statistical distribution of the LCP code. For example, texture contrast is calculated using the standard deviation of pixel grayscale values ​​in the local region; uniformity is calculated by statistically analyzing the proportion of patterns where the number of 0-to-1 or 1-to-0 transitions does not exceed two; and directionality is calculated by calculating the angle corresponding to the maximum peak value in the local gradient direction histogram. It is understandable that micro-texture features capture subtle structural changes and grayscale distribution patterns on the solder joint surface.

[0032] In the specific implementation, the geometric morphological features and micro-texture features extracted from the same solder joint region at different scales are serially concatenated and standardized. Serial concatenation connects the geometric morphological feature vectors from each layer of the blurred image pyramid with the micro-texture feature vectors from the original independent image regions in sequence into a long feature vector. For example, if geometric morphological features are extracted from three scales, each scale containing four features, and three micro-texture features are extracted from the original image, the dimension of the long feature vector is fifteen. Standardization uses the Z-score standardization method to adjust the long feature vector to eliminate the influence of feature dimensions and numerical range. Optionally, standardization can be based on online calculated moving averages and standard deviations to adapt to different batches of solder joint images. The mean vector and standard deviation vector are statistically obtained from a large number of solder joint image samples during the training phase. A comprehensive feature descriptor for the solder joint region is generated, and the comprehensive feature descriptors of all solder joint regions constitute a comprehensive feature descriptor set.

[0033] In one embodiment of this application, the training process of the pre-trained deep classification network includes collecting image samples containing various PCBA solder joint defect types to construct a training dataset. Each sample in the training dataset contains a solder joint region image and its corresponding true defect type label. The various PCBA solder joint defect types include cold solder joints, solder bridges, insufficient solder, excessive solder, and normal solder joints. Each solder joint region image corresponds to a true defect type label, which is generated by manual annotation or automated detection equipment. In some embodiments, the construction of the training dataset needs to consider the balance of sample numbers for different defect types. Data augmentation techniques such as rotation, translation, scaling, or adding noise can be used to increase the sample number of a few categories, thereby avoiding class imbalance during the deep classification network training process. The initial deep classification network is iteratively trained using the training dataset. In each iteration, the cross-entropy loss between the network's predicted defect type and the true defect type label is calculated. The cross-entropy loss function measures the difference between the predicted probability distribution of the deep classification network and the true label distribution. Optionally, the cross-entropy loss function can be combined with weight parameters to assign different penalty weights to different defect categories to address potential class imbalance. The cross-entropy loss function is calculated using the following formula: .in, This represents the cross-entropy loss value. This represents the total number of defect categories. The label indicating the actual defect type is in the category. The uniquely encoded components on, This represents the output layer of a deep classification network in terms of category. The predicted probability is calculated. In practice, the gradient descent algorithm is used to update the weight parameters of the deep classification network to minimize the cross-entropy loss. The gradient descent algorithm calculates the gradient of the cross-entropy loss relative to the weight parameters of each layer of the deep classification network and updates the weights along the negative gradient direction until the classification accuracy of the network on the validation dataset converges to a preset threshold, thus obtaining a pre-trained deep classification network. It can be understood that the gradient descent algorithm can use stochastic gradient descent, momentum method, or Adam optimizer to accelerate the training process and improve stability.

[0034] In implementation, the pre-trained deep classification network comprises multiple fully connected layers and a Softmax output layer. The fully connected layers are sequentially connected, each containing several neurons and equipped with a non-linear activation function such as ReLU. The number of neurons in the Softmax output layer equals the total number of defect categories. During detection, the comprehensive feature descriptor of each solder joint region is used as input to the deep classification network, and this comprehensive feature descriptor is input as a vector to the first fully connected layer. Non-linear transformations and feature dimensionality reduction are then performed sequentially through multiple fully connected layers. Each fully connected layer performs a linear weighted summation of the input vector and applies a non-linear activation function. Feature dimensionality reduction is achieved by progressively reducing the number of neurons, ultimately mapping the high-dimensional comprehensive feature descriptor to a low-dimensional feature vector. The Softmax output layer maps the dimensionality-reduced feature vector to a probability distribution belonging to different defect types. The Softmax output layer performs a linear transformation on the low-dimensional feature vector and applies the Softmax function to convert the output value into a probability value. Defect types include cold solder joints, solder bridges, insufficient solder, excessive solder, and normal solder joints. In some embodiments, the number of fully connected layers and the number of neurons in each layer can be configured according to the dimension of the comprehensive feature descriptor and the complexity of the classification task. For example, when the comprehensive feature descriptor is a 15-dimensional vector, the deep classification network can be configured to contain two fully connected layers, with the first fully connected layer having ten neurons and the second fully connected layer having five neurons. It can be understood that the nonlinear activation function introduces a nonlinear transformation, enabling the deep classification network to learn complex feature mapping relationships. The defect type with the highest probability is taken as the initial defect classification result, and the corresponding probability value is used as the confidence score. For example, if the Softmax output layer of the deep classification network calculates the probability distribution for a certain solder joint area as follows: 0.05 for poor solder joint, 0.80 for bridging, 0.10 for insufficient solder, 0.03 for excessive solder, and 0.02 for normal solder joint, then bridging is selected as the initial defect classification result, and its corresponding probability value of 0.80 is used as the confidence score.

[0035] See Figure 3This is a graph showing the loss variation of a PCBA solder joint defect detection model during training. Both training and validation losses decrease rapidly from high values, stabilizing after approximately 30 iterations and eventually converging within the 0–0.5 range, indicating that the model effectively learned the data features. The training loss consistently remained lower than the validation loss, consistent with typical deep learning training patterns. The training and validation losses maintained similar trends, without significant overfitting ("training loss continuously decreasing, validation loss rising"), indicating that the current training configuration effectively controlled overfitting. If different network structures, feature dimensions, or optimizers are subsequently tried, the effectiveness of the improved solutions can be quantitatively evaluated by comparing the loss curves, providing data support for technological iteration. If anomalies appear in the loss curve during subsequent model iterations or changes in data distribution, the problem can be quickly identified, ensuring the stability of the detection system.

[0036] In one embodiment of this application, a spatial association model across solder joints is constructed based on the initial defect classification results and confidence scores. A solder joint adjacency graph is constructed based on the physical location coordinates of the solder joints in a high-resolution surface image. Nodes in the solder joint adjacency graph represent solder joints, and edges represent the adjacency relationships between solder joints. The physical location coordinates of the nodes are obtained through solder joint region localization processing. The rule for establishing edges can be defined as establishing an edge between the nodes representing the two solder joints when the Euclidean distance between the center points of two solder joints is less than a preset adjacency threshold. In some embodiments, the adjacency threshold can be set based on the typical solder joint spacing on the PCBA board. For example, the solder joint spacing is typically between 1 mm and 3 mm, and the corresponding adjacency threshold in the image pixel coordinate system can be set to 20 pixels. The solder joint adjacency graph is stored and represented using an adjacency matrix or adjacency list data structure. In the weld joint adjacency graph, each node is assigned an initial defect classification result and a confidence score as an initial state. Each node stores a state vector, which contains the node's current defect type label and confidence score. During initialization, the node's current defect type label is equal to the initial defect classification result obtained from the deep classification network, and the node's confidence score is directly adopted from the confidence score output by the deep classification network.

[0037] In practical implementation, a defect propagation energy function is defined. Based on the assumption that adjacent solder joints have similar defect types, the function calculates the inconsistency cost between the current node state and the states of adjacent nodes. For any edge in the solder joint adjacency graph, a node inconsistency term is defined, which is proportional to the difference in defect type labels between two adjacent nodes. For any node in the solder joint adjacency graph, a node data fidelity term is defined, which is inversely proportional to the difference between the corresponding node's defect type label and its initial defect classification result, and weighted by the corresponding node's confidence score. The defect propagation energy function is the sum of the node data fidelity terms of all nodes and the node inconsistency terms of all edges. The node inconsistency term can be defined as zero cost if two adjacent nodes have the same defect type label, otherwise a positive constant. The node data fidelity term can be defined as a penalty incurred when the node's current defect type label is inconsistent with the initial defect classification result; the penalty strength is inversely proportional to the node's confidence score. The mathematical expression of the defect propagation energy function is: .in, This represents the total energy value of the defect propagation energy function. This represents the set of all nodes in the solder joint adjacency graph. This represents a node in the solder joint adjacency diagram. Represents a node Node data fidelity items, This represents the set of all edges in the weld point adjacency graph. Indicates the connection node and nodes One of the edges, Representing an edge Inconsistencies between nodes, The regularization parameter represents the weights of the data fidelity items at the balancing nodes and the inconsistencies between nodes.

[0038] In some embodiments, referring to Table 1, the specific calculation of the node data fidelity item is shown, illustrating an example of the calculation of the node data fidelity item at a certain moment during the optimization iteration process for the three nodes.

[0039] Table 1: Example Table of Node Data Fidelity Calculation

[0040]

[0041] Optionally, the definition of inconsistencies between nodes can be further refined. For example, different cost weights can be assigned to different label differences based on the correlation between defect types in the process. The cost difference between bridging and excessive solder can be less than the cost difference between bridging and normal solder joints. In specific implementation, a graph optimization algorithm is used to minimize the defect propagation energy function. The graph optimization algorithm can use an iterative conditional pattern, which sequentially traverses each node in the solder joint adjacency graph and evaluates all possible defect type labels, selecting the one that minimizes the defect propagation energy function. The label with the largest decrease in value is taken as the new current defect type label for the node. The defect type label of each node is iteratively updated until convergence. The convergence condition can be the defect propagation energy function. The value no longer decreases or the labels of all nodes no longer change. It can be understood that the iterative conditional mode is a local optimization method that seeks a local optimum of the defect propagation energy function by iteratively updating the labels of individual nodes. The defect classification state of each node after convergence is taken as the optimized defect classification result, and the defect classification state of each node is its final determined defect type label. Optionally, the graph optimization algorithm can also use a graph cut-based global optimization algorithm to minimize the defect propagation energy function.

[0042] See Figure 4 This is a scatter plot showing the distribution of node data fidelity items during the construction phase of a cross-solder joint spatial association model. It illustrates the calculated fidelity item values ​​for 10 solder joint nodes during the optimization process, and their relationship with defect type and confidence level. The magnitude of the fidelity item values ​​clearly presents the "data fidelity penalty" incurred by each solder joint during optimization. A value of 0 indicates complete trust in the initial classification, while a larger value indicates a higher cost for deviating from the initial result. Nodes with a fidelity item value of 0 are more likely to maintain the initial classification result during graph optimization. Nodes with higher fidelity item values ​​are more susceptible to the influence of defect types from adjacent nodes and are corrected during iterations to satisfy the spatial consistency assumption. This plot visually demonstrates the distribution characteristics of node data fidelity items and the potential correlation between defect type and fidelity item values, providing a basis for model parameter tuning.

[0043] In one embodiment of this application, defective solder joint areas are located based on optimized defect classification results. The optimized defect classification results are compared with preset normal solder joint type labels, where the preset normal solder joint type label is a predefined string or enumeration value such as "Normal". All solder joint areas whose defect type is not a normal solder joint label are filtered out. For example, the optimized defect classification results may include five types: "cold solder joint", "bridging solder", "insufficient solder", "excessive solder", and "Normal". By performing string matching or non-equality checks on the optimized defect classification result of each solder joint against "Normal", all solder joint areas classified as "cold solder joint", "bridging solder", "insufficient solder", or "excessive solder" can be identified. In some embodiments, the preset normal solder joint type labels can be stored in a configuration file or database table, allowing the comparison logic to flexibly adapt to different label naming conventions. The filtering operation traverses all nodes in the solder joint adjacency graph, reads the optimized defect classification result of each node, and compares it with the preset normal solder joint type label. Nodes that do not match are marked as defective solder joint areas.

[0044] In practice, based on the coordinates of each independent image region recorded during the solder joint area localization process in the original high-resolution surface image, the bounding box of each defective solder joint region in the image is determined. The solder joint area localization process synchronously records the coordinates of each region in the original high-resolution surface image when generating independent image regions. These coordinates can be the vertex coordinates of the smallest bounding rectangle of the solder joint contour closed curve in the original image coordinate system. For each solder joint region marked as defective, its corresponding coordinates are retrieved and used to define the bounding box. The bounding box is typically represented by the coordinates of the top-left and bottom-right vertices of a rectangle. For example, if the top-left corner of the smallest bounding rectangle of a defective solder joint region is (x1, y1) and the bottom-right corner is (x2, y2), then the bounding box of that solder joint is defined by these two coordinates. It is understood that the format for recording coordinates must be consistent with the format for subsequent annotation generation to ensure accurate localization. For each defective solder joint region, a label entry is generated. This entry includes a text description of the defect type and the coordinates of its bounding box. The defect type description is directly taken from the optimized defect classification result for that region, and the bounding box coordinates are the previously determined vertex coordinates. Label entries can be organized using structured formats such as JSON, XML, or plain text. For example, a label entry could be represented as "Solder Joint ID: 5, Defect Type: Solder Bridging, Bounding Box: [x1, y1, x2, y2]". Optionally, the label entry may also include initial confidence scores obtained from the deep classification network or other intermediate information from the optimization process for review and reference.

[0045] In practice, all generated annotation entries are overlaid onto the original high-resolution surface image to form a visual defect detection report image. The overlay operation is achieved by drawing graphics and text on a copy of the original high-resolution surface image. For each annotation entry, a rectangle is drawn at the corresponding position in the image based on its bounding box coordinates. The color and line width of the rectangle can be differentiated according to the defect type to enhance readability. Simultaneously, a text description of the defect type is annotated near or inside the rectangle. The font, size, and color of the text can be configured to ensure clear visibility. In some embodiments, the overlay process can be completed using drawing functions provided by image processing libraries such as OpenCV and PIL. After all annotation entries are drawn, the final visual defect detection report image is generated. This image visually displays the original PCBA board surface image and the location and type information of all detected defective solder joints.

[0046] See Figure 5 This is a comparison chart of defect identification accuracy before and after optimization of the spatial correlation model in the PCBA solder joint defect detection process. At all iterations, the accuracy after optimization was significantly higher than before, consistently remaining above 98%, demonstrating the value of the spatial correlation model in improving detection reliability. The accuracy before optimization increased linearly with the number of iterations, while the accuracy after optimization reached a high level in the early stages of iteration, with subsequent growth slowing down, indicating rapid model convergence and higher iteration efficiency. The optimized curve exhibits less fluctuation, indicating that spatial consistency constraints effectively reduce the impact of single-point misclassification, making the detection results more stable and reliable. Through spatial correlation analysis, misclassification of isolated solder joints was effectively corrected, improving the overall accuracy by approximately 0.7-3.7 percentage points. The optimized model converges faster, reducing unnecessary computational overhead and improving detection efficiency.

[0047] The above are merely preferred embodiments of this application and are not intended to limit this application in any other way. Any person skilled in the art may use the above-disclosed technical content to make changes or modifications to equivalent embodiments and apply them to other fields. However, any simple modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of this application without departing from the content of the technical solution of this application shall still fall within the protection scope of the technical solution of this application.

Claims

1. A PCBA solder joint defect detection method based on image recognition, characterized in that, The method includes: Acquire a high-resolution surface image of the PCBA board to be inspected, the high-resolution surface image containing complete optical information of multiple solder joint areas; The high-resolution surface image is processed to locate the solder joint region, and the independent image region corresponding to each solder joint in the high-resolution surface image is identified and segmented to generate a set of solder joint region images. Multi-scale feature extraction processing is performed on each independent image region in the set of solder joint region images to obtain a comprehensive feature descriptor set containing macroscopic morphological features and microscopic texture features; The comprehensive feature descriptor set is input into a pre-trained deep classification network for defect pattern matching analysis to generate initial defect classification results and confidence scores for each solder joint area. Based on the initial defect classification results and confidence scores, a cross-solder joint spatial association model is constructed to analyze the defect association characteristics between adjacent solder joint regions. Spatial consistency optimization is then performed on the initial defect classification results to generate optimized defect classification results, including: Based on the physical location coordinates of the solder joints in the high-resolution surface image, a solder joint adjacency graph is constructed, where nodes represent solder joints and edges represent the adjacency relationships between solder joints. In the weld joint adjacency diagram, each node is assigned the initial defect classification result and the confidence score as an initial state. Define a defect propagation energy function, which is based on the assumption that adjacent solder joints have similar defect types, and calculates the inconsistency cost between the current node state and the adjacent node states, including: For any edge in the weld joint adjacency graph, define an inconsistency term between nodes, which is proportional to the degree of difference in the defect type labels corresponding to two adjacent nodes. For any node in the weld joint adjacency graph, a node data fidelity item is defined. The node data fidelity item is inversely proportional to the difference between the defect type label of the corresponding node and its initial defect classification result, and is weighted by the confidence score of the corresponding node. The defect propagation energy function is the sum of the node data fidelity terms of all nodes and the node inconsistency terms of all edges; The defect propagation energy function is minimized using a graph optimization algorithm, and the defect classification state of each node is iteratively updated until convergence. The defect classification status of each node after convergence is taken as the optimized defect classification result. Based on the optimized defect classification results, the defective solder joint areas are located, and annotation information containing the defect type and location coordinates is generated on the original high-resolution surface image.

2. The PCBA solder joint defect detection method based on image recognition according to claim 1, characterized in that, The high-resolution surface image is processed to locate solder joint regions, identify and segment the independent image region corresponding to each solder joint in the high-resolution surface image, and generate a set of solder joint region images, including: The high-resolution surface image is processed using an edge detection algorithm to extract candidate edge information for all weld point contours in the image; Based on prior knowledge of the geometric shape of the solder joint contour, the candidate edge information is subjected to shape matching and screening to obtain an accurate closed curve of the solder joint contour. Based on the closed curve of the solder joint outline, the pixel region corresponding to each solder joint is cropped from the high-resolution surface image using image masking technology to form the independent image region; All independent image regions are normalized to ensure that each solder joint region image has the same pixel size, forming the solder joint region image set.

3. The PCBA solder joint defect detection method based on image recognition according to claim 2, characterized in that, Multi-scale feature extraction processing is performed on each independent image region in the set of solder joint region images to obtain a comprehensive feature descriptor set containing macroscopic morphological features and microscopic texture features, including: Multi-scale blurring is performed by applying Gaussian filters of different sizes to each independent image region, generating blurred image pyramids corresponding to different scale spaces. At each layer of the blurred image pyramid, the geometric features of the solder joint area are extracted, including area, perimeter, roundness, and aspect ratio of the minimum bounding rectangle. Simultaneously, a local binary mode algorithm is applied to each independent image region to extract the micro-texture features of the solder joint surface, including texture contrast, uniformity, and directionality. The geometric features and microtexture features extracted from the same solder joint area at different scales are serially spliced ​​and standardized to generate the comprehensive feature descriptor of the solder joint area. The comprehensive feature descriptors of all solder joint areas constitute the comprehensive feature descriptor set.

4. The PCBA solder joint defect detection method based on image recognition according to claim 3, characterized in that, The comprehensive feature descriptor set is input into a pre-trained deep classification network for defect pattern matching analysis, generating initial defect classification results and confidence scores for each solder joint region, including: The pre-trained deep classification network contains multiple fully connected layers and a Softmax output layer; The comprehensive feature descriptor set is used as the input of the deep classification network, and nonlinear transformation and feature dimensionality reduction are performed sequentially through the multiple fully connected layers; In the Softmax output layer, the dimensionality-reduced feature vectors are mapped to probability distributions belonging to different defect types, including cold solder joints, solder bridging, insufficient solder, excessive solder, and normal solder joints. The defect type with the highest probability is taken as the initial defect classification result, and the corresponding probability value is taken as the confidence score.

5. The PCBA solder joint defect detection method based on image recognition according to claim 4, characterized in that, Based on the optimized defect classification results, the defective solder joint areas are located, and annotation information containing defect type and location coordinates is generated on the original high-resolution surface image, including: The optimized defect classification results are compared with the preset normal solder joint type labels to filter out all solder joint areas whose defect type is not a normal solder joint. Based on the coordinate position of each independent image region recorded during the solder joint area localization process in the original high-resolution surface image, the location bounding box of each defective solder joint area in the image is determined. For each defective solder joint area, a label entry is generated. The label entry contains a text description of the defect type of the solder joint area and the coordinate data of its location bounding box. All generated annotation entries are overlaid onto the original high-resolution surface image to form a visual defect detection report image.

6. The PCBA solder joint defect detection method based on image recognition according to claim 5, characterized in that, After acquiring the high-resolution surface image of the PCBA board to be inspected, the method further includes: The high-resolution surface image is preprocessed, including illumination equalization and background noise suppression. The illumination equalization process calculates the global and local histograms of the image and applies histogram specification techniques to make the overall illumination distribution of the image uniform. The background noise suppression process employs a nonlocal mean denoising algorithm, which smooths noise in the image background area while preserving the details of the solder joint edges.

7. The PCBA solder joint defect detection method based on image recognition according to claim 6, characterized in that, The training process of the pre-trained deep classification network includes: Collect image samples containing various PCBA solder joint defect types to construct a training dataset. Each sample in the training dataset contains an image of the solder joint area and its corresponding real defect type label. The initialized deep classification network is iteratively trained using the training dataset. In each iteration, the cross-entropy loss between the defect type predicted by the network and the actual defect type label is calculated. The weight parameters of the deep classification network are updated using the gradient descent algorithm to minimize the cross-entropy loss until the classification accuracy of the network on the validation dataset converges to a preset threshold, thus obtaining the pre-trained deep classification network.

8. A PCBA solder joint defect detection system based on image recognition, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the image recognition-based PCBA solder joint defect detection method according to any one of claims 1 to 7.

Images