A BGA solder joint segmentation method based on UNet deep separable residual network
By improving the architecture of the UNet deep separable residual network, the speed and robustness issues of traditional BGA solder joint segmentation methods in complex backgrounds are solved, achieving efficient solder joint segmentation with reduced parameters and improved segmentation accuracy.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- INTELLIGENT LOCK RES INST OF WENZHOU UNIV
- Filing Date
- 2022-11-08
- Publication Date
- 2026-08-04
AI Technical Summary
Existing traditional BGA solder joint segmentation methods are slow and have poor robustness in complex backgrounds, while deep learning methods suffer from parameter redundancy and slow computation speed.
We employ a UNet-based deep separable residual network. By improving the architecture of the UNet model, we replace standard convolutions with deep separable convolutions, add a Batch Normalization layer, introduce residual structures in the encoder, and optimize the training parameters and network structure.
It significantly reduced model complexity, improved computational efficiency, and enabled rapid segmentation of BGA solder joints while maintaining high accuracy. The number of parameters was only 12.17% of the original model. The crossover ratio, accuracy, and F1 score were improved by 2.17%, 0.52%, and 1.18%, respectively, and the FPS reached 114.8 frames per second.
Smart Images
Figure CN115937099B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of BGA solder joint segmentation, and more specifically to a BGA solder joint segmentation method based on a UNet depth separable residual network. Background Technology
[0002] Currently, with the rapid development of the electronics and information industry, high-density integration has become the main development trend of electronic devices. Ball Grid Array (BGA) is a mainstream packaging technology widely used in integrated circuit boards. Its main feature is the use of solder balls instead of leads on the bottom of the chip, resulting in smaller packaged chips, more pins, and better electrical performance. Since the BGA solder joints are located on the bottom of the chip, X-ray imaging systems are generally used to inspect the soldering quality. Common BGA soldering defects include short circuits, voids, cold solder joints, and solder joints that are too large or too small. The ability to accurately segment the solder joints affects the diagnosis of solder joint defects; therefore, research on solder joint segmentation methods is crucial.
[0003] Currently, traditional image processing methods are widely used. Traditional BGA solder joint segmentation methods are based on edge, threshold, morphological and geometric features. These methods have good results for segmenting solder joints with clear outlines and simple backgrounds. However, for complex backgrounds, multiple methods are often required, resulting in slow segmentation speed and poor robustness.
[0004] In recent years, with the development of artificial intelligence technology, deep learning has been widely used due to its excellent feature selection and extraction capabilities. Scholars have applied it to the field of BGA solder joint segmentation. For example, Chen Yaqiong's 2021 master's thesis, "Research on BGA Solder Ball Void Defect Detection Based on Deep Learning," and Zhao Ruixiang et al.'s 2019 paper, "Bubble Segmentation of Ball Grid Array Solder Ball Edges Combining Fully Convolutional Networks and K-means Clustering," both publicly use fully convolutional neural networks to segment BGA solder joint images, achieving good accuracy, but with redundant parameters and slow network computation speed. Summary of the Invention
[0005] To address the shortcomings of existing technologies, the present invention aims to provide a BGA solder joint segmentation method based on a UNet depth separable residual network.
[0006] To achieve the above objectives, the present invention provides the following technical solution:
[0007] A method for segmenting BGA solder joints based on a UNet deep separable residual network includes the following steps:
[0008] 1) Obtain the BGA solder joint dataset;
[0009] 2) Preprocess the acquired dataset;
[0010] 3) Construct a UNet model based on a deep separable residual network, and improve the architecture of the UNet model.
[0011] 3.1) Replace the standard convolutions in the encoder and decoder of the UNet model with depthwise separable convolutions;
[0012] 3.2) Change the feature extraction structure of the third, fourth and fifth layers of the encoder from the original two convolutional layers to three convolutional layers, with the convolutional kernel parameters remaining unchanged. The first convolution of the third and fourth layers increases the number of channels, while the second and third convolutions do not change the number of channels. All convolutions in the fifth layer do not change the number of channels.
[0013] 3.3) Add a Batch Normalization layer after the convolutional layers of the encoder and decoder;
[0014] 3.4) Add residual structures to each layer at the encoder end, and use 1×1 convolutional layers to solve the problem of mismatch between the input and output dimensions of the residual block. Each layer is added at the beginning of the first convolutional kernel and the end of the last convolutional kernel, and finally output after passing through the non-linear activation function ReLU.
[0015] 4) Train the network by feeding the preprocessed dataset into the improved UNet model;
[0016] 5) Input the BGA solder joint image to be segmented into the trained improved UNet network to output the segmentation result.
[0017] In step 1), the dataset acquisition involves capturing images of the BGA solder joints on the PCB board using a microfocus X-Ray device and downloading some BGA solder joint images from a search engine as the raw dataset.
[0018] In step 2), the original data is augmented by randomly rotating, flipping, and scaling the Augmentor data augmentation library. After filtering, several images are obtained as the original dataset, and then randomly divided into training and validation sets according to a set ratio.
[0019] The UNet network consists of two parts: an encoder and a decoder. Each part contains five layers. The first four layers of the decoder each contain two 3×3 convolutions and a ReLU activation function. Then, they are downsampled by max pooling before entering the next layer. The fifth layer is upsampled by bilinear interpolation after convolution and activation. Each layer of the decoder contains two 3×3 convolutions and a ReLU activation function. The output of the previous layer is concatenated with the corresponding encoder layer and then convolved and activated before entering the next layer. Finally, the decoder passes through a convolutional layer with two channels to achieve foreground and background segmentation.
[0020] In step 3.1), the standard 3×3 convolution is split into a 3×3 convolution with the same number of input channels and a 1×1 convolution with the same number of output channels, and channel-wise convolution and point-wise convolution operations are performed respectively.
[0021] The specific training parameters for the model in step 4) are as follows: using the Adam optimizer, the initial learning rate is set to 0.0001, the number of training iterations is set to 200, the learning rate is dynamically adjusted in an exponential decay manner according to the number of iterations, and the minimum learning rate is set to 0.000001.
[0022] In step 5), the parameter corresponding to the minimum loss value within the training period is selected as the final weight to segment the BGA solder joint image.
[0023] A processor for running a program, wherein the program executes any one of the above-described BGA solder joint segmentation methods based on a UNet depth-separable residual network.
[0024] The beneficial effects of this invention are as follows: Based on the UNet model, this invention significantly reduces redundant parameters, lowers model complexity, and improves computational efficiency by employing depthwise separable convolutions. The addition of batch normalization layers improves data distribution and accelerates network convergence. By modifying the encoder structure and increasing the number of convolutional layers, the feature learning capability is enhanced, enabling the model to learn more semantic information from deeper networks. Furthermore, the introduction of residual networks into each layer of the encoder's feature extraction solves the problems of gradient vanishing, gradient exploding, and network degradation during backpropagation. Experimental results show that the proposed BGA solder joint segmentation method based on the UNet depthwise separable residual network has only 12.17% of the parameters of the original model. The intersection-over-union ratio, accuracy, and F1 score reach 92.4%, 98.31%, and 96.05%, respectively, representing improvements of 2.17%, 0.52%, and 1.18% compared to the original UNet network. The FPS reaches 114.8 frames per second, achieving both rapid BGA solder joint segmentation and high accuracy. Attached Figure Description
[0025] Figure 1 This is a schematic diagram of the process of the present invention;
[0026] Figure 2 This invention provides an improved UNet network model.
[0027] Figure 3 This is a structural diagram of the residual module used in this invention;
[0028] Figure 4 This is a schematic diagram of the depthwise separable convolution principle used in this invention;
[0029] Figure 5This is a comparison chart of the segmentation effects of the model used in this invention with other models;
[0030] Figure 6 This is the final segmentation result diagram of the model used in this invention. Detailed Implementation
[0031] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.
[0032] It should be noted that all directional indications (such as up, down, left, right, front, back, etc.) in the embodiments of the present invention are only used to explain the relative positional relationship and movement of each component in a certain specific posture (as shown in the figure). If the specific posture changes, the directional indication will also change accordingly.
[0033] In this invention, unless otherwise explicitly specified and limited, the terms "connection," "fixed," etc., should be interpreted broadly. For example, "fixed" can mean a fixed connection, a detachable connection, or an integral part; it can mean a mechanical connection or a connection; it can mean a direct connection or an indirect connection through an intermediate medium; it can mean the internal communication of two elements or the interaction between two elements, unless otherwise explicitly limited. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.
[0034] The hardware environment used for implementation is configured as follows: Windows 10 64-bit operating system, AMD Ryzen 55600H CPU 3.30GHz, GeForce GTX 1650 GPU, and 16GB of RAM; the software environment is configured as follows: Python 3.10, PyTorch 1.11.0 and CUDA 11.3 deep learning frameworks.
[0035] As shown in the figure, this invention discloses a BGA solder joint segmentation method based on a UNet depth-separable residual network, which includes the following steps:
[0036] 1) Obtain the BGA solder joint dataset. The dataset is obtained by acquiring images of the BGA solder joints on the PCB board using a microfocus X-Ray device, and by downloading some BGA solder joint images from a search engine as the raw dataset.
[0037] This invention uses LabelMe software to annotate solder joint images, with the background portion having a value of 0 and the solder joint area having a value of 1, generating a JSON file, which is then converted into a PNG grayscale image using code.
[0038] 2) The acquired dataset is preprocessed. The original data is augmented by random rotation, flipping, scaling and other operations using the Augmentor data augmentation library. After filtering, 400 images are obtained as the original dataset, and they are randomly divided into training set and validation set in an 8:2 ratio.
[0039] 3) Construct a UNet model based on a deep separable residual network, and improve the architecture of the UNet model.
[0040] The UNet network consists of two parts: an encoder and a decoder. Each part has five layers. The first four layers of the decoder each contain two 3×3 convolutions and a ReLU activation function. Then, they are downsampled by max pooling before entering the next layer. The fifth layer is upsampled by bilinear interpolation after convolution and activation. Each layer of the decoder contains two 3×3 convolutions and a ReLU activation function. The output of the previous layer is concatenated with the corresponding encoder layer and then convolved and activated before entering the next layer. Finally, the decoder passes through a convolutional layer with two channels to achieve foreground and background segmentation.
[0041] The specific improvement steps are as follows:
[0042] 3.1) Replace standard convolutions in the encoder and decoder with depthwise separable convolutions, such as... Figure 4 As shown, the specific operation is as follows: the standard 3×3 convolution is split into a 3×3 convolution with the same number of input channels and a 1×1 convolution with the same number of output channels, and channel-wise convolution and point-wise convolution operations are performed respectively. The number of parameters of the standard convolution is 3×3×3×2, while the number of parameters of the depthwise separable convolution is only 3×3×1×3+1×1×3×2, which greatly reduces the number of parameters.
[0043] 3.2) Change the feature extraction structure of the third, fourth and fifth layers of the encoder from the original two convolutional layers to three convolutional layers, with the convolutional kernel parameters remaining unchanged. The first convolution of the third and fourth layers increases the number of channels, while the second and third convolutions do not change the number of channels. All convolutions in the fifth layer do not change the number of channels.
[0044] 3.3) Add a Batch Normalization (BN) layer after the convolutional layers of the encoder and decoder;
[0045] 3.4) Add a residual structure to each layer at the encoder end, such as a residual block. Figure 3As shown, a 1×1 convolutional layer and a batch normalization (BN) layer are used to solve the problem of input and output dimension mismatch in residual blocks, such as... Figure 2 The DR_2Conv blocks are added at the beginning of the first convolutional kernel and the end of the last convolutional kernel in each layer, and finally output after passing through the non-linear activation function ReLU.
[0046] The relationship between input and output can be expressed as H(x) = relu(F(x,w) + Zx), where relu is a nonlinear activation function, x is the input of the residual block, w is the weight parameter, Z is the mapping matrix, F(x,w) is the output after the first layer of linear transformation and activation, and H(x) is the output of the residual block.
[0047] Figure 2 In the diagram, DR_2Conv indicates that this layer uses depthwise separable convolutional and residual structures, and the 2 indicates that two convolutional layers are used. Similarly, DR_3Conv indicates that this module uses three convolutional layers. The Up_Cat module is a feature concatenation structure, which mainly concatenates the upsampled deep features with the features learned by the encoder along the channel dimension.
[0048] Figure 2 The improved UNet network shown takes a 512×512 pixel image with 3 channels as input. After each convolutional layer of the encoder, the number of output channels is 64, 128, 256, and 512, respectively, with the resolution halved after each downsampling. After each convolutional layer of the decoder, the number of output channels is 512, 356, 128, and 64, respectively, with the resolution doubling after each upsampling. The image is then decoded to be the same size as the input.
[0049] 4) The preprocessed dataset is fed into the improved UNet model to train the network. The specific training parameters are as follows: the Adam optimizer is used, the initial learning rate is set to 0.0001, the number of training iterations is set to 200, the learning rate is dynamically adjusted exponentially based on the number of iterations, the minimum learning rate is set to 0.000001, and the binary cross-entropy function is used as the loss function for training the model. The specific formula is as follows: Where y is a binary label 0 or 1, p(y) is the probability that the output belongs to the label y, and N represents the total number of training iterations.
[0050] 5) Input the BGA solder joint image to be segmented into the trained improved UNet network to output the segmentation result, select the parameter corresponding to the minimum loss value within the training period as the final weight, and segment the BGA solder joint image.
[0051] Furthermore, this invention uses a confusion matrix to statistically analyze the classification results, and the confusion matrix of the classification results is shown in the table.
[0052]
[0053] Among them, TP (True Positive) indicates that pixels belonging to solder joints in the image are correctly classified; FP (False Positive) indicates that non-solder joint areas (i.e., background) in the image are misclassified as solder joints; FN (False Negative) indicates that solder joint areas in the image are misclassified as background; and TN (True Negative) indicates that non-solder joint areas in the image are correctly classified.
[0054] Furthermore, based on the confusion matrix, this experiment uses three metrics to evaluate network performance: Intersection over Union (IoU), Accuracy (Acc), and F1 Score (F1_Score). A higher score indicates better network segmentation and a superior model. The specific formulas are as follows:
[0055]
[0056]
[0057]
[0058] Where precision represents the ratio of correctly predicted solder joints to all correctly predicted solder joints; recall represents the ratio of correctly predicted solder joints to the total number of actual solder joints, and the specific formulas are as follows:
[0059]
[0060]
[0061] Furthermore, mainstream semantic segmentation algorithms UNet, DeepLabV3+, and PSPNet are selected as comparison algorithms for the method of this invention. Figure 5 This is a comparison chart showing the segmentation effects of different network algorithms. Figure 5 (a) is the original image, which includes several typical cases such as simple background, complex background with vias and RC interference, solder joints being obscured, and solder joints having large air bubbles inside. Figure 5 (b) is the label image corresponding to the original image. Figure 5 (c)- Figure 5 (f) shows the segmentation results obtained by the mainstream segmentation algorithms PSPNet, DeepLabV3+, UNet, and the method presented in this paper, respectively. The red portion represents the solder joint area, and the black portion represents the background. Figure 5 (c) and Figure 5 (d) It can be seen that although both PSPNet and DeepLabV3+ can extract the approximate outline of BGA solder joints, the edge information is relatively coarse. Figure 5(e) It can be observed that although the original UNet network can extract the edge information of the solder joints well, it suffers from false detections (first image). It cannot extract a complete contour when the solder joint is occluded (second image), and when there are air bubbles inside the solder joint, the bubble area is classified as background, failing to extract the entire solder joint. The method proposed in this invention can extract BGA solder joints better under various conditions. The table below compares the segmentation evaluation metrics of different network models on a self-made dataset. It has fewer parameters and lower computational cost, and compared to other algorithms, it shows significant improvements in intersection-over-union ratio, accuracy, and F1 score, while achieving an FPS of 114.8 frames / second, ensuring high accuracy while rapidly extracting solder joints.
[0062]
[0063] Figure 6 The image shown is an example of using the method described in this paper to map the segmented result back to the original image and extract the segmented BGA solder joint image.
[0064] Embodiments of the present invention also provide a computer-readable storage medium. Optionally, in this embodiment, the storage medium can be used to store the program code executed by the BGA solder joint segmentation method based on UNet depth-separable residual network provided in the above embodiments.
[0065] The aforementioned storage medium can be located in any computer terminal in a group of computer terminals in a computer network, or in any mobile terminal in a group of mobile terminals.
[0066] The storage medium is configured to store program code for performing the steps disclosed in the invention above, in order to divide the BGA solder joints.
[0067] The present invention also discloses a processor for running a program, wherein the program executes any one of the above-described BGA solder joint segmentation methods based on UNet depth separable residual networks.
[0068] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the BGA solder joint segmentation method and apparatus based on the UNet deep separable residual network proposed in this invention. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby realizing the image segmentation method described above. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to the segmentation system via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.
[0069] The processor can access the information and application programs stored in the memory via the transmission device to execute the above steps, thereby performing the segmentation process on the BGA solder joints.
[0070] Those skilled in the art will understand that all or part of the steps in the above methods can be implemented by a program instructing the hardware of the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.
[0071] The embodiments should not be regarded as limitations on the present invention, but any improvements made based on the spirit of the present invention should be within the protection scope of the present invention.
Claims
1. A BGA joint segmentation method based on UNet deep separable residual network, characterized by: It includes the following steps: 1) Obtain the BGA solder joint dataset; 2) Preprocess the acquired dataset; 3) Construct a UNet model based on a deep separable residual network, and improve the architecture of the UNet model. 3.1) Replace the standard convolutions in the encoder and decoder of the UNet model with depthwise separable convolutions; 3.2) Change the feature extraction structure of the third, fourth and fifth layers of the encoder from the original two convolutional layers to three convolutional layers, with the convolutional kernel parameters remaining unchanged. The first convolution of the third and fourth layers increases the number of channels, while the second and third convolutions do not change the number of channels. All convolutions in the fifth layer do not change the number of channels. 3.3) Add a Batch Normalization layer after the convolutional layers of the encoder and decoder; 3.4) Add residual structures to each layer at the encoder end, and use 1×1 convolutional layers to solve the problem of mismatch between the input and output dimensions of the residual block. Each layer is added at the beginning of the first convolutional kernel and the end of the last convolutional kernel, and finally output after passing through the non-linear activation function ReLU. 4) Train the network by feeding the preprocessed dataset into the improved UNet model; 5) Input the BGA solder joint image to be segmented into the trained improved UNet network to output the segmentation result. The UNet network consists of two parts: an encoder and a decoder. Each part contains five layers. The first four layers of the decoder each contain two 3×3 convolutions and a ReLU activation function. Then, they are downsampled by max pooling before entering the next layer. The fifth layer is upsampled by bilinear interpolation after convolution and activation. Each layer of the decoder contains two 3×3 convolutions and a ReLU activation function. The output of the previous layer is concatenated with the corresponding encoder layer and then convolved and activated before entering the next layer. Finally, the decoder passes through a convolutional layer with two channels to achieve foreground and background segmentation.
2. The BGA joint segmentation method based on UNet deep separable residual network according to claim 1, characterized in that: In step 1), the dataset acquisition involves capturing images of the BGA solder joints on the PCB board using a microfocus X-Ray device and downloading some BGA solder joint images from a search engine as the raw dataset.
3. The BGA joint segmentation method based on UNet deep separable residual network according to claim 1, characterized in that: In step 2), the original data is augmented by randomly rotating, flipping, and scaling the Augmentor data augmentation library. After filtering, several images are obtained as the original dataset, and then randomly divided into training and validation sets according to a set ratio.
4. The BGA joint segmentation method based on UNet deep separable residual network according to claim 1, characterized in that: In step 3.1), the standard 3×3 convolution is split into a 3×3 convolution with the same number of input channels and a 1×1 convolution with the same number of output channels, and channel-wise convolution and point-wise convolution operations are performed respectively.
5. The BGA joint segmentation method based on UNet deep separable residual network according to claim 1, characterized in that: The specific training parameters for the model in step 4) are as follows: using the Adam optimizer, the initial learning rate is set to 0.0001, the number of training iterations is set to 200, the learning rate is dynamically adjusted in an exponential decay manner according to the number of iterations, and the minimum learning rate is set to 0.000001.
6. The BGA joint segmentation method based on UNet deep separable residual network according to claim 1, characterized in that: In step 5), the parameter corresponding to the minimum loss value within the training period is selected as the final weight to segment the BGA solder joint image.
7. A processor, comprising: The processor is configured to run a program, and the program performs the BGA solder joint segmentation method based on the UNet deep separable residual network according to any one of claims 1 to 6 when the program is running.