BGA welding spot segmentation method and system based on X-ray imaging

Abstract

The invention provides a BGA welding spot segmentation method and system based on X-ray imaging, and belongs to the technical field of image recognition. The BGA welding spot segmentation method based on X-ray imaging comprises the following steps: image acquisition: acquiring an original BGA welding spot imaging image; image preprocessing: preprocessing the BGA original welding spot imaging image to form a BGA welding spot diagram; the feature information of the welding spots is extracted from the BGA welding spot diagram, the welding spots are positioned, preliminary segmentation is achieved, and separation of any welding spot and the background is achieved; and fine segmentation: automatically adjusting parameters of a Hough transform circle detection algorithm by using a deep convolutional neural network, and then carrying out feature learning on a coarse segmentation result by using the Hough transform circle detection algorithm capable of automatically adjusting the parameters so as to realize accurate segmentation of the BGA welding spots. The method has the beneficial effects that the precision of automatic segmentation is improved, and the industrial production requirements of precision, efficiency and the like are effectively met.

Description

Technical Field

[0001] This invention relates to the field of image recognition technology, and specifically to a method and system for segmenting BGA solder joints based on X-ray imaging. Background Technology

[0002] With the continuous development of modern electronic equipment and the increasing speed of industrial production, the performance requirements for PCB boards are constantly rising, leading to the rapid development of BGA (Ball Grid Array) packaging technology. Chips packaged using BGA are smaller, have more pins, larger pin pitch, and superior performance, making this type of packaging widely used across various industries. However, in actual production, the input / output terminals of BGA packages are arrayed on the bottom of the chip using circular or columnar solder joints. The internal structure of these solder joints is invisible, making quality inspection of the solder joints after the BGA-packaged chip is soldered onto the PCB board extremely challenging. Furthermore, if the PCB board's functionality is not tested before the product is released to the market, leading to problems during use, engineers will need to inspect each component individually, consuming significant manpower and resources. Therefore, quality inspection of devices using BGA packaging technology is a problem that urgently needs to be solved.

[0003] Currently, the main methods for inspecting the quality of BGA solder joints are:

[0004] (1) Visual inspection

[0005] Visual inspection is usually the first step in BGA soldering failure detection and analysis. The main method involves using the human eye, or with the aid of a simple optical magnifying glass or industrial microscope, to inspect the solder joints at the edges of the BGA device, looking for any anomalies visible to the naked eye, such as cold solder joints, cracked solder joints, misaligned solder joints, and warping. However, visual inspection can only detect solder balls at the device edges and cannot detect defects in the solder joints inside the package. In industrial production, BGA solder joints are usually hidden, therefore visual inspection is not suitable for such highly integrated and demanding products.

[0006] (2) Staining detection

[0007] Dyeing inspection is a destructive testing technique. Its principle involves first penetrating a highly penetrating dye into the cracked area of ​​the solder joint at high temperature, then vacuum drying it, and finally using mechanical stress to pull open the BGA solder package to observe the degree and distribution of cracks inside the solder joint. This allows for direct observation of all information regarding the solder interface quality. However, dyeing inspection is not only time-consuming and places high demands on the testing equipment and devices, but it also causes irreversible damage to the solder joint, resulting in resource waste.

[0008] (3) Metallographic sections

[0009] Metallographic cross-section inspection is also a destructive testing technique. The BGA sample is first encapsulated in resin, then sanded to the desired location, and finally polished to obtain a high-quality cross-section for observation of its microstructure. However, this method is not only cumbersome and complex, prone to errors leading to inaccurate results, but it can also cause irreversible damage to solder joints.

[0010] Destructive testing techniques cause irreversible damage to chips, have high testing costs, and long testing cycles. Currently, this method is rarely used for abnormal detection of BGA solder joints.

[0011] (4) Boundary Scan Detection

[0012] Boundary scan inspection (BSI) is a non-destructive testing technique. The basic idea is to add a shift register unit near the chip's input / output pins. When the chip is in debug mode, the BSI isolates the chip from its external inputs / outputs. Through the BSI unit, the chip's input / output signals can be controlled and observed, allowing for monitoring of the chip's continuity. However, using BSI to inspect BGA solder joint quality involves analyzing the chip circuitry and checking for continuity. This process requires highly specialized knowledge from the inspectors, thus having significant limitations.

[0013] (5) X-ray detection

[0014] X-ray inspection is a non-destructive testing technique. X-rays have strong penetrating power. When an X-ray beam scans an object of a certain thickness, the detector converts the received X-rays into visible light, which is then converted into an electrical signal via photoelectric conversion. Finally, an analog-to-digital converter generates a digital signal, which is transmitted to a computer for processing and ultimately forms an image. This technique is called X-ray imaging. It is commonly used in quality inspection and safety testing for non-destructive testing of the internal structure of materials. Utilizing the transmission characteristics of X-rays to inspect BGA solder joints is not only simple to operate and detects the solder ball bonding hidden at the bottom of the device, but also does not damage the solder joints, making it the most effective method for BGA solder joint quality inspection currently available.

[0015] While existing methods can inspect BGA solder joint quality, they suffer from numerous drawbacks, including long inspection cycles, cumbersome procedures, high costs, difficulties in manual operation, and the inability to quickly and accurately locate abnormal solder joints. Furthermore, X-ray-based BGA solder joint imaging exhibits low contrast and significant background interference, placing high demands on inspectors and resulting in a high false positive rate if relying solely on visual observation. Given the significant limitations of current X-ray imaging-based BGA solder joint inspection methods, there is an urgent need for an X-ray imaging-based BGA solder joint segmentation method that can overcome the shortcomings of low contrast and high background interference, effectively segmenting and locating BGA solder joints, thereby improving the accuracy of BGA solder joint inspection. Summary of the Invention

[0016] To address the problems of low contrast and significant background interference in BGA solder joint imaging in existing technologies, which prevent effective segmentation and positioning of BGA solder joints, this invention provides a BGA solder joint segmentation method and system based on X-ray imaging, thereby improving the accuracy of BGA solder joint positioning and realizing automated solder joint detection based on X-rays.

[0017] The present invention provides a BGA solder joint segmentation method based on X-ray imaging, comprising the following steps:

[0018] S1: Image Acquisition: Acquire raw solder joint images of the BGA;

[0019] S2: Image preprocessing: Preprocess the original BGA solder joint imaging image to form a BGA solder joint image;

[0020] S3: Coarse segmentation: Extract the feature information of the solder joints from the BGA solder joint image, locate the solder joints and achieve preliminary segmentation, so as to separate any solder joint from the background;

[0021] S4: Fine segmentation: The parameters of the Hough transform circle detection algorithm are automatically adjusted using a deep convolutional neural network. Then, the Hough transform circle detection algorithm with automatically adjusted parameters is used to learn features from the coarse segmentation results to achieve accurate segmentation of BGA solder joints.

[0022] Furthermore, in step S1, an X-ray inspection device is used to acquire images of the original BGA solder joints. The X-ray beam is tilted at a certain angle and aimed at the substrate with the chip to be inspected, so that the device pads on the substrate with the chip to be inspected do not completely overlap with the PCB pads.

[0023] The X-ray detection device includes an X-ray generator disposed on one side of the chip to be detected, an X-ray detector disposed on the other side of the chip to be detected for acquiring X-ray signals, and an image processing system connected to the output end of the X-ray detector. The original BGA solder joints acquired by the image processing system include solder ball rings and PCB pad rings and device pad rings inside the solder ball rings, wherein the PCB pad rings and device pad rings are nested together.

[0024] Furthermore, the tilt angle of the X-ray is 30-75 degrees.

[0025] Furthermore, in step S2, the image preprocessing method is as follows:

[0026] S201: Use annotation tools to annotate the original BGA solder joint imaging image to separate the BGA solder joint area from the background area;

[0027] S202: Normalize the solder joint image, the normalization process including:

[0028] Image normalization step: The pixel values ​​of the image are transformed to form solder joint areas and background areas with obvious differences in pixel values;

[0029] Data normalization steps: The label data is normalized using the zero-mean standardization method;

[0030] Neural network normalization step: The batch normalization strategy (BN) is adopted to remap the input distribution of the neural network layers to a certain interval, thereby avoiding gradient vanishing and accelerating the convergence speed of the model.

[0031] S203: Perform feature enhancement on the BGA solder joint image to obtain a BGA solder joint image with more obvious solder joint features.

[0032] Furthermore, in the neural network normalization step,

[0033] During model training, a normalization calculation is performed once for each channel. The calculation range for each layer is [Batch, 1, Height, width], and the number of calculations is equal to the number of channels. The BN calculation steps are as follows:

[0034] (1) Calculate the mean within a batch of layer l. The original input is X = [x(1), x(2), ..., x(m)], and the hidden values ​​of layer l are [z(1), z(2), ..., z(m)], where i is the channel number. The formula for calculating the mean μ is:

[0035]

[0036] (2) Calculate the variance, σ2 :

[0037]

[0038] (3) For each channel Perform normalization separately:

[0039]

[0040] The above formula transforms the original data distribution into a normal distribution, then The mean becomes 0 and the variance becomes 1, where ∈ takes a small number to prevent the expression from being divided by 0 when the variance is 0.

[0041] (4) Scale and translate to obtain the translated data.

[0042]

[0043] Here, γ and β are training hyperparameters, the purpose of which is to prevent the data from having a linear distribution and to adjust it to appropriate values ​​by continuously learning from the dataset.

[0044] Furthermore, in step S203, the BGA solder joint image is processed by a Butterworth filter to improve the clarity of the image details, and then histogram equalization is applied to enhance the contrast of the BGA solder joint image, making the solder joint features more obvious.

[0045] Furthermore, in step S3, an improved U-Net neural network model is used to perform coarse segmentation of the BGA solder joint image. The improved U-Net neural network model includes a channel-wise convolution module and a point-wise convolution module.

[0046] The channel-wise convolution module includes an input channel and a first filter set at the output end of the input channel, which is the same as the number of input channels. The data of one input channel is processed by convolution through the corresponding first filter to obtain an output feature map with the same number of input channels.

[0047] The pointwise convolution module has several convolution kernels, each with a size of 1×1×M, where M is the number of input channels. Each convolution kernel has a second filter with a size of 1×1×M at its output.

[0048] Further, in step S4, the deep convolutional neural network includes a convolutional pooling module and a fully connected layer module disposed at the output end of the convolutional pooling module. The convolutional pooling module includes several convolutional modules, and each convolutional module output is provided with a max pooling layer module. A ReLU layer is provided between the convolutional modules and the max pooling layer modules. The output end of the fully connected layer module is the output of the deep convolutional neural network.

[0049] This invention also provides a BGA solder joint segmentation system based on X-ray imaging, used to implement the aforementioned BGA solder joint segmentation method based on X-ray imaging, comprising:

[0050] Image acquisition module: used to acquire raw solder joint images of BGA;

[0051] Image preprocessing module: used to preprocess the original BGA solder joint imaging image to form a BGA solder joint image;

[0052] Coarse segmentation module: Used to extract feature information of solder joints from BGA solder joint image, locate solder joints and perform preliminary segmentation, and achieve separation of arbitrary solder joints from the background;

[0053] Fine segmentation module: It is used to automatically adjust the parameters of the Hough transform circle detection algorithm using a deep convolutional neural network, and then use the automatically adjusted parameter Hough transform circle detection algorithm to learn features from the coarse segmentation results to achieve accurate segmentation of BGA solder joints.

[0054] Compared with existing technologies, the beneficial effects of this invention are: by combining deep learning segmentation methods with BGA solder joint segmentation, it achieves the localization and segmentation of BGA solder joints by training a BGA solder joint segmentation model without distinguishing between the shape of the BGA solder joints and the complexity of the background. Compared with existing X-ray segmentation methods, this invention effectively improves segmentation accuracy and efficiency, thus effectively meeting production requirements such as accuracy and efficiency within a reasonable error range.

[0055] The Hough transform circle detection algorithm can locate solder joint regions in coarsely segmented BGA solder joint images and remove burrs and noise from the coarse segmentation. However, due to its poor generalization ability, different parameters need to be set for solder joints captured in different environments to achieve accurate segmentation. Therefore, this invention combines the Hough transform circle detection algorithm with a deep convolutional neural network to construct a network model with automatically adjusted parameters, thereby automating solder joint segmentation.

[0056] By tilting the X-ray at a certain angle, the ring formed by the solder ball will not block the ring formed by the PCB pad substrate, thus making it easy to detect the soldering status of the solder ball. Attached Figure Description

[0057] To more clearly illustrate the solutions in this invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0058] Figure 1 This is a flowchart of the method of the present invention;

[0059] Figure 2 Schematic diagram of BGA solder joint structure;

[0060] Figure 3 A schematic diagram showing the X-ray setup position of the X-ray inspection device and the projection of the solder balls.

[0061] Figure 4 This is a schematic diagram of the histogram equalization process of the present invention;

[0062] Figure 5 A schematic diagram showing the effect of preprocessing the original BGA solder joint imaging image;

[0063] Figure 6 This is a schematic diagram of the original U-Net network convolution operation logic;

[0064] Figure 7 This is a schematic diagram of the convolution operation logic of the improved U-Net network of the present invention;

[0065] Figure 8 A schematic diagram of the BGA solder joint segmentation network structure;

[0066] Figure 9 This is a schematic diagram of the segmentation results after coarse segmentation of each network;

[0067] Figure 10 This is a schematic diagram of the fine-segmentation network model structure. Detailed Implementation

[0068] Unless otherwise defined, all technical and scientific terms used in this invention have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains; the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention; the terms "comprising" and "having," and any variations thereof, in the specification, claims, and foregoing drawings are intended to cover non-exclusive inclusion. The terms "first," "second," etc., in the specification, claims, or foregoing drawings are used to distinguish different objects, not to describe a particular order.

[0069] In this invention, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment to other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this invention can be combined with other embodiments.

[0070] To enable those skilled in the art to better understand the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings.

[0071] like Figure 1 As shown, this invention provides a BGA solder joint segmentation method based on X-ray imaging, thereby improving the accuracy of BGA solder joint positioning and realizing automated solder joint detection based on X-rays.

[0072] The present invention provides a BGA solder joint segmentation method based on X-ray imaging, comprising the following steps:

[0073] S1: Image Acquisition: Acquire raw solder joint images of the BGA;

[0074] S2: Image preprocessing: Preprocess the original BGA solder joint imaging image to form a BGA solder joint image;

[0075] S3: Coarse segmentation: Extract the feature information of the solder joints from the BGA solder joint image, locate the solder joints and achieve preliminary segmentation, so as to separate any solder joint from the background;

[0076] S4: Fine segmentation: The parameters of the Hough transform circle detection algorithm are automatically adjusted using a deep convolutional neural network. Then, the Hough transform circle detection algorithm with automatically adjusted parameters is used to learn features from the coarse segmentation results to achieve accurate segmentation of BGA solder joints.

[0077] The following provides a detailed explanation of each step.

[0078] Step S1: Image acquisition.

[0079] For BGA solder joints, the absorption of X-ray radiation energy varies depending on the density of the area, resulting in different imaging results. Areas that absorb less X-ray radiation energy will have brighter images, while areas that absorb more will have darker images. Therefore, the BGA solder joint area in X-ray imaging will appear darker than the surrounding area. The advantages of this X-ray imaging method are its real-time performance and high efficiency, as well as the fact that it does not damage the object being tested and can store the imaging results for a long time. It is currently the most commonly used imaging method for non-destructive testing.

[0080] BGA-packaged I / O terminals are encapsulated on the bottom of the chip in an array of circular solder joints. The internal structure is not visible to the outside. This imaging method allows visualization of the internal structure of the BGA solder joints, enabling the detection of soldering anomalies. A diagram of the BGA solder joint structure is shown below. Figure 2 As shown.

[0081] The left side shows a cross-sectional view of a real BGA solder joint, and the right side shows a structural diagram of a BGA solder joint. Each solder joint consists of five parts: the component, the component pad, the solder ball, the PCB pad, and the PCB substrate. The solder ball must be connected to both the component pad and the PCB pad simultaneously to ensure that the solder joint is conductive, meaning the soldering is normal and it can function properly. If any of these components is missing from the BGA solder joint, or if the solder ball is not connected to the upper and lower pads, it indicates that the solder joint is not conductive, meaning the soldering is abnormal and it cannot function properly. The soldering process of attaching the chip to the PCB board is highly susceptible to interference from various external or human factors, such as poor plating on the PCB board surface, inappropriate solder wettability, improper reflow soldering temperature settings, and inadequate moisture protection measures on the PCB board. These abnormalities can lead to soldering abnormalities in the BGA solder joints, preventing them from functioning properly.

[0082] Common X-ray inspection systems project X-rays perpendicularly onto the object being inspected. Because different regions within a BGA solder joint have varying thicknesses, they absorb X-ray radiation at different intensities, theoretically resulting in three concentric rings in the image. However, with existing inspection systems, the diameter of the BGA solder ball is larger than the diameter of the PCB base pad. The rings formed by the PCB base pads are obscured by the rings formed by the solder ball, causing the three rings to overlap. This makes it impossible to detect the specific solder joint details based on the imaging results. Therefore, this invention modifies the acquisition device to obtain clearer solder joint information. The modified structure is as follows: Figure 3 As shown.

[0083] The X-ray detection device includes an X-ray generator disposed on one side of the chip to be detected, an X-ray detector disposed on the other side of the chip to be detected for acquiring X-ray signals, and an image processing system connected to the output end of the X-ray detector. In this invention, the X-rays emitted by the X-ray generator are tilted at a certain angle (the tilt angle can be 30-75 degrees) and aimed at the chip to be tested. The ring formed by the solder balls will not block the ring formed by the PCB pad substrate. Therefore, if the solder joint is well soldered, three nested and regular rings will be formed. If the solder joint is abnormal, three nested rings cannot be formed.

[0084] Step S2: Image preprocessing.

[0085] Since the number and radius of BGA solder joints in images captured in different scenarios vary, segmentation becomes challenging. Therefore, a segmentation preprocessing step is required to separate the solder joint region from the background region in the obtained raw BGA solder joint image.

[0086] The image preprocessing method in this example is as follows:

[0087] S201: Use the Labelme data annotation tool to annotate the original BGA solder joint imaging image. The annotated solder joint image after Labelme annotation is shown, where the white part corresponds to the solder joint area and the black part corresponds to the background area.

[0088] S202: Normalize the solder joint image.

[0089] S2021: Image Normalization Steps

[0090] When collecting BGA solder joint datasets, variations in operators, operating environments, and imaging settings can lead to significant differences in the X-ray appearance of solder joints on different PCBs. Therefore, to prevent experimental errors caused by inconsistencies in BGA solder joint image appearance, this invention employs image normalization. The idea is to convert all image datasets into the same standard form and use image invariant moments to find a set of parameters that can eliminate the influence of image differences, thus resisting the impact of image transformation on the results. Image normalization transforms the pixel values ​​of the image, ensuring that the pixel values ​​in the solder joint area are similar, the pixel values ​​in the background area are similar, and the differences between the two remain. The most common processing method is maximum-minimum normalization, and the normalized value norm is defined as:

[0091]

[0092] Where xi represents the pixel value of the image, max(x) represents the maximum value in the image matrix, and min(x) represents the minimum value in the image matrix. After normalization, the pixel values ​​of the image are all in the range [0,1].

[0093] S2022: Data Normalization Steps

[0094] The fine segmentation step, which uses the Hough transform circle detection algorithm, requires manual adjustment of parameters based on the radius and spacing of the weld point images. Therefore, to automate this parameter adjustment process, this invention utilizes a deep convolutional neural network to learn features from the segmentation results obtained in the coarse segmentation step. The parameters to be adjusted serve as learning labels. However, since the parameters are on different orders of magnitude with significant numerical differences, if they are not aligned to the same order of magnitude, finding the optimal solution during neural network learning becomes extremely difficult, resulting in low model accuracy. Therefore, to eliminate the errors introduced by the differences in the order of magnitude of the dataset and to accelerate model convergence, this invention normalizes the label data in this step using z-score (zero-mean) normalization.

[0095] S2023: Neural Network Normalization Steps

[0096] Both the coarse segmentation step and the fine segmentation step of this invention are based on deep convolutional neural networks to learn the characteristics of BGA solder joints. As the number of network layers increases, the input distribution of deeper layers in the network changes more significantly. In order to reduce the error caused by the change in input distribution to the experiment, this invention adopts the Batch Normalization (BN) normalization strategy when designing the network. The purpose is to remap the input distribution of the network layers to a certain interval, avoid gradient vanishing, and also accelerate the convergence speed of the model.

[0097] During model training, a normalization calculation is performed once for each channel. The calculation range for each layer is [Batch, 1, Height, width], and the number of calculations is equal to the number of channels. The BN calculation steps are as follows:

[0098] (1) Calculate the mean within a batch of layer l. The original input is X = [x(1), x(2), ..., x(m)], and the hidden values ​​of layer l are [z(1), z(2), ..., z(m)], where i is the channel number. The formula for calculating the mean μ is:

[0099]

[0100] (2) Calculate the variance, σ 2 :

[0101]

[0102] (3) For each channel Perform normalization separately:

[0103]

[0104] The above formula transforms the original data distribution into a normal distribution, then The mean becomes 0 and the variance becomes 1, where ∈ takes a small number to prevent the expression from being divided by 0 when the variance is 0.

[0105] (4) Scale and translate to obtain the translated data.

[0106]

[0107] Here, γ and β are training hyperparameters, the purpose of which is to prevent the data from having a linear distribution and to adjust it to appropriate values ​​by continuously learning from the dataset.

[0108] S203: Perform feature enhancement on the BGA solder joint image to obtain a BGA solder joint image with more obvious solder joint features.

[0109] In step S203, the BGA solder joint image is processed by a Butterworth filter to improve the clarity of the image details, and then histogram equalization is used to enhance the contrast of the BGA solder joint image, making the solder joint features more obvious.

[0110] Butterworth low-pass filters (BLPF) are an image enhancement method based on transform domain operations. This invention uses BLPF to preprocess BGA solder joints.

[0111] like Figure 4 As shown, because the contrast of BGA solder joint images obtained by X-ray imaging is low, this invention further enhances the contrast of the solder joint images after applying a Butterworth low-pass filter using histogram equalization, making the solder joint features clearer. The image of the BGA solder joint image after image enhancement is shown below. Figure 5 As shown, the left side displays the original BGA solder joints and histogram, while the right side displays the enhanced BGA solder joints and histogram. Figure 5 As can be seen, the characteristics of the BGA solder joint (circular portion) are enhanced by this method.

[0112] Since the background complexity of PCB board BGA solder joint images based on X-ray imaging is relatively high, the high background complexity has a significant impact on the segmentation of the foreground and background. In addition, the shape of BGA solder joints is relatively complex. Therefore, in order to achieve higher segmentation accuracy, this invention adopts a deep learning segmentation method for the preprocessed image, which divides the segmentation process into two steps: coarse segmentation and fine segmentation.

[0113] S3: Coarse segmentation: By using a deep convolutional neural network to learn about BGA solder joints, an improved U-Net network is proposed. This network directly extracts the feature information of the solder joints from the BGA solder joint image, locates the solder joints, and achieves preliminary segmentation, thus separating any solder joint from the background.

[0114] The original U-Net network's U-shaped structure and skip connections allow for more efficient utilization of image feature information. Therefore, this invention retains these two advantages and proposes an improved U-Net network. The original U-Net's convolutional layers are conventional convolutional layers, and the computational logic is as follows: Figure 6 As shown.

[0115] In conventional convolution, each filter is responsible for all input channels. Therefore, the number of channels in each filter matches the number of input channels. Corresponding channels are convolved with their corresponding input channels, and the results are summed to obtain the output. Thus, each filter yields only one output feature map, and the number of output feature maps equals the number of filters. As shown in the diagram, with 3 input channels, each filter also has 3 channels. Since there are 4 filters in this process, there will be 4 output feature maps. Therefore, during training, the number of parameters required for each convolutional layer is: P = c0 × (k w ×k h ×c i ), where c0 represents the number of output channels, c i k represents the number of input channels. w k represents the kernel width. h This indicates the height of the convolution kernel.

[0116] To achieve a BGA solder joint segmentation model with fewer parameters, shorter training time, and better model performance, the improved U-Net proposed in this invention replaces conventional convolution with depthwise separable convolution. The computational logic of depthwise separable convolution is as follows: Figure 7 As shown.

[0117] The depthwise separable convolution feature extraction process of this invention consists of two steps: channel-wise convolution (executed within the purple box) and pointwise convolution (executed within the blue box). In channel-wise convolution, each filter only needs to handle one input channel, so the filter's channel count is 1. Each input channel is convolved by only one filter, eliminating the need for convolution with all filters. Therefore, this process yields an output feature map with the same number of input channels, without any expansion of the output feature map. Thus, the parameter count for the channel-wise convolution step is: P1 = c0 × k w ×k h Where c0 represents the number of output channels, k w k represents the kernel width. h This indicates the height of the convolution kernel. In pointwise convolution, the convolution process is similar to that of regular convolution. Each convolution kernel has a size of 1×1×M, where M is the number of input channels corresponding to the kernel. The number of 1×1×M filters in this process determines the number of output feature maps. The parameters for this pointwise convolution step are: P2 = c0×c i Where c0 represents the number of output channels, c i This represents the number of input channels when calculating the number of parameters. Therefore, the total number of parameters required for depthwise separable convolution is: P = P1 + P2 = c0 × (k w ×k h +c iAs can be seen from the above, for the same input and output, compared with conventional convolution, the depthwise separable convolution of this invention requires fewer parameters, less computation, and has a shorter model execution time.

[0118] This invention, based on an X-ray imaging BGA solder joint segmentation model, is an improved version of the U-Net network, with the network structure as follows: Figure 8 As shown, each convolutional kernel corresponds to a feature map, while in the original U-Net, each convolutional kernel must be convolved with the feature map of each input channel. Changing the convolutional layers in the original U-Net network to depthwise separable convolutions can effectively save learning parameters.

[0119] To verify the effectiveness and superiority of the improved U-Net network in BGA solder joint segmentation, the coarse segmentation step of this invention compares the original U-Net network with the improved U-Net network. Furthermore, the number of channels in both the original and improved U-Net networks is changed, and the parameters of each network are statistically analyzed. The networks and their parameters are shown in Table 1.

[0120] Table 1. Network and its parameters

[0121]

[0122] It can be seen that the improved U-Net has fewer parameters than the original U-Net. The more initial convolutional channels there are, the more significant the difference becomes. Fewer parameters result in a smaller model size and less computation. Models of the above-mentioned network types were trained using 4500 sets of X-ray imaging solder joint data and corresponding labeled data as training samples, and 450 sets as test samples, with a sample size of 502×320. The trained segmentation model was then used to perform segmentation prediction on BGA solder joint images based on X-ray imaging. The segmentation results are shown below. Figure 9 As shown.

[0123] The segmentation results show that each network fails to accurately segment regions where the background and solder joint colors are similar, and also fails to accurately segment their edge regions, resulting in burrs at the edges. Figure 9 As can be seen, the improved U-Net with an initial convolutional number of 96 achieves the most accurate edge segmentation. Compared with the edge segmentation results of the U-Net network with an initial convolutional number of 96, the segmentation results of the U-Net with an initial convolutional number of 96 are coarser and confuse some background areas with solder joint areas.

[0124] To quantitatively evaluate the performance of the improved U-Net segmentation network model, experimental evaluation methods (execution time ET, mean intersection-over-union ratio MIoU, mean pixel accuracy MPA, and Dice coefficient) were applied to evaluate each network model. The experimental results are shown in Table 2.

[0125] Table 2. Results of coarse segmentation comparison experiment

[0126]

[0127]

[0128] As shown above, the more initial convolutions there are, the longer the segmentation prediction takes. Meanwhile, the improved U-Net network executes faster than the original U-Net network. Therefore, the network structure used in this study saves more time. Furthermore, the improved U-Net network with 96 initial convolution channels achieves an average intersection-over-union (MIoU) of 94.14%, an average pixel accuracy (MPA) of 97.37%, and a Dice coefficient of 97.01%. Its MIoU is 1.5% higher than the original U-Net with 96 initial convolution channels, its MPA is 0.45% higher, and its Dice coefficient is 0.94% higher. Moreover, the improved U-Net network with 96 initial convolution channels also outperforms other networks in terms of MIoU, MPA, and Dice coefficient. Therefore, the improved U-Net network with 96 initial convolution channels achieves the best segmentation performance, better detail processing, and higher segmentation accuracy.

[0129] S4: Fine segmentation: The parameters of the Hough transform circle detection algorithm are automatically adjusted using a deep convolutional neural network. Then, the Hough transform circle detection algorithm with automatically adjusted parameters is used to learn features from the coarse segmentation results to achieve accurate segmentation of BGA solder joints.

[0130] For fine segmentation of BGA solder joints, the key is to address the impact of different imaging environments on BGA solder joint imaging. Different imaging results will lead to different coarse segmentation results, thus affecting the parameters of the Hough transform circle detection algorithm. The main physical parameters affecting the segmentation result are the maximum circle radius, minimum circle radius, minimum circle spacing, and intersection point threshold.

[0131] The deep convolutional neural network model structure used for fine segmentation in this invention is as follows: Figure 10As shown, the deep convolutional neural network includes a convolutional pooling module and a fully connected layer module disposed at the output of the convolutional pooling module. The convolutional pooling module comprises several convolutional layers, and each convolutional layer has a max-pooling layer at its output. A ReLU layer is disposed between the convolutional modules and the max-pooling layer, and a max-pooling layer is connected using the ReLU non-linear activation function. The output of the fully connected layer module is the output of the deep convolutional neural network. In this example, the convolutional kernel size is 3×3 with a stride of 1, the max-pooling layer size is 2×2 with a stride of 2, and a fully connected layer is connected to the model at the end as the network output.

[0132] The input image to the model is the segmentation result from the coarse segmentation step, and the labels are the corresponding Hough transform circle detection algorithm parameters. The fine segmentation model structure and parameters are explored experimentally, focusing on the influence of factors such as the number of convolutional layers, the number of convolutional kernels, and the learning rate set during training on the performance of the BGA solder joint fine segmentation model. The output parameter values ​​of the model are determined based on experimental data.

[0133] This invention primarily utilizes a deep convolutional neural network (CNN) to learn the parameters of the Hough transform, achieving numerical prediction and accurate solder joint segmentation. Therefore, this invention introduces mean squared error (MSE) to evaluate the numerical prediction model and uses the coarse segmentation evaluation metrics, mean intersection-over-union (MIoU) and mean pixel accuracy (MPA), to evaluate the fine segmentation performance. Experimental verification shows that when the number of convolutional modules during training of the deep convolutional neural network in this example is 3 layers, the initial number of convolutional channels is 32, and the learning rate is 0.001, the performance of each metric is optimal. These values ​​are then set as the parameters for the deep convolutional neural network model of this invention. By setting the deep convolutional neural network model to learn the parameters of the Hough transform circle detection algorithm, the Hough transform circle detection algorithm can achieve more accurate positioning and segmentation of BGA solder joints.

[0134] This invention also provides a BGA solder joint segmentation system based on X-ray imaging, used to implement the aforementioned BGA solder joint segmentation method based on X-ray imaging, comprising:

[0135] Image acquisition module: used to acquire raw solder joint images of BGA;

[0136] Image preprocessing module: used to preprocess the original BGA solder joint imaging image to form a BGA solder joint image;

[0137] Coarse segmentation module: Used to extract feature information of solder joints from BGA solder joint image, locate solder joints and perform preliminary segmentation, and achieve separation of arbitrary solder joints from the background;

[0138] Fine segmentation module: It is used to automatically adjust the parameters of the Hough transform circle detection algorithm using a deep convolutional neural network, and then use the automatically adjusted parameter Hough transform circle detection algorithm to learn features from the coarse segmentation results to achieve accurate segmentation of BGA solder joints.

[0139] Experimental verification

[0140] To highlight the superior performance of the segmentation method of this invention, three experiments were designed: manual segmentation, coarse segmentation only, and a combination of coarse and fine segmentation. The same BGA solder joint image was segmented, and the three segmentation methods were evaluated by quantitative evaluation indicators such as segmentation speed and segmentation accuracy MIoU. The results are shown in Table 3.

[0141] Table 3 Performance analysis of different segmentation methods

[0142]

[0143] Table 3 of the experimental results shows that the manual segmentation method has the highest accuracy, with an MIoU of 99.06%, meeting the requirement of greater than 95% segmentation accuracy. However, this method has the slowest segmentation speed, taking 300 seconds to segment one solder joint image, far from meeting the requirement of less than 60 seconds. The method with only coarse segmentation has the fastest segmentation speed, taking only 2.28 seconds to segment one solder joint image, meeting the requirement of less than 60 seconds. However, this method has the lowest segmentation accuracy, with an MIoU of only 94.02%, failing to meet the requirement of greater than 95% segmentation accuracy. The method combining coarse and fine segmentation has a segmentation speed of 2.55 seconds and a segmentation accuracy MIoU of 97.42%. This method can meet both the segmentation time and accuracy requirements. Compared with the segmentation time of less than 60 seconds, the segmentation efficiency of this method is significantly improved, which can far meet the actual inspection needs. The segmentation accuracy of 97.42% can also meet the requirement of solder joint integrity, making it the most appropriate segmentation method.

[0144] Compared with existing technologies, the beneficial effects of this invention are: by combining deep learning segmentation methods with BGA solder joint segmentation, it achieves the localization and segmentation of BGA solder joints by training a BGA solder joint segmentation model without distinguishing between the shape of the BGA solder joints and the complexity of the background. Compared with existing X-ray segmentation methods, this invention effectively improves segmentation accuracy and efficiency, thus effectively meeting production requirements such as accuracy and efficiency within a reasonable error range.

[0145] The Hough transform circle detection algorithm can locate solder joint regions in coarsely segmented BGA solder joint images and remove burrs and noise from the coarse segmentation. However, due to its poor generalization ability, different parameters need to be set for solder joints captured in different environments to achieve accurate segmentation. Therefore, this invention combines the Hough transform circle detection algorithm with a deep convolutional neural network to construct a network model with automatically adjusted parameters, thereby automating solder joint segmentation.

[0146] By tilting the X-ray at a certain angle, the ring formed by the solder ball will not block the ring formed by the PCB pad substrate, thus making it easy to detect the soldering status of the solder ball.

[0147] The specific embodiments described above are preferred embodiments of the present invention and are not intended to limit the specific scope of the present invention. The scope of the present invention includes, but is not limited to, these specific embodiments. All equivalent changes made in accordance with the present invention are within the protection scope of the present invention.

Claims

1. A method for segmenting BGA solder joints based on X-ray imaging, characterized in that, Includes the following steps: S1: Image Acquisition: Acquire raw solder joint images of the BGA; S2: Image preprocessing: Preprocess the original BGA solder joint imaging image to form a BGA solder joint image; S3: Coarse segmentation: Extract the feature information of the solder joints from the BGA solder joint image, locate the solder joints and achieve preliminary segmentation, so as to separate any solder joint from the background; S4: Fine segmentation: The parameters of the Hough transform circle detection algorithm are automatically adjusted using a deep convolutional neural network. Then, the Hough transform circle detection algorithm with automatically adjusted parameters is used to learn features from the coarse segmentation results to achieve accurate segmentation of BGA solder joints.

2. The BGA solder joint segmentation method based on X-ray imaging according to claim 1, characterized in that: In step S1, an X-ray inspection device is used to acquire images of the original BGA solder joints. The X-ray beam is tilted at a certain angle and aimed at the substrate with the chip to be inspected, so that the device pads on the substrate with the chip to be inspected do not completely overlap with the PCB pads. The X-ray detection device includes an X-ray generator disposed on one side of the chip to be detected, an X-ray detector disposed on the other side of the chip to be detected for acquiring X-ray signals, and an image processing system connected to the output end of the X-ray detector. The original BGA solder joints acquired by the image processing system include solder ball rings and PCB pad rings and device pad rings inside the solder ball rings, wherein the PCB pad rings and device pad rings are nested together.

3. The BGA solder joint segmentation method based on X-ray imaging according to claim 2, characterized in that: The tilt angle of the X-rays is 30-75 degrees.

4. The BGA solder joint segmentation method based on X-ray imaging according to claim 1, characterized in that: In step S2, the image preprocessing method is as follows: S201: Use annotation tools to annotate the original BGA solder joint imaging image to separate the BGA solder joint area from the background area; S202: Normalize the solder joint image, the normalization process including: Image normalization step: The pixel values ​​of the image are transformed to form solder joint areas and background areas with obvious differences in pixel values; Data normalization steps: The label data is normalized using the zero-mean standardization method; Neural network normalization step: The batch normalization strategy (BN) is adopted to remap the input distribution of the neural network layers to a certain interval, thereby avoiding gradient vanishing and accelerating the convergence speed of the model. S203: Perform feature enhancement on the BGA solder joint image to obtain a BGA solder joint image with more obvious solder joint features.

5. The BGA solder joint segmentation method based on X-ray imaging according to claim 4, characterized in that: In the neural network normalization step, During model training, a normalization calculation is performed once for each channel. The calculation range for each layer is [Batch, 1, Height, width], and the number of calculations is equal to the number of channels. The BN calculation steps are as follows: (1) Calculate the mean within a batch of layer l. The original input is X = [x(1), x(2), ..., x(m)], and the hidden values ​​of layer l are [z(1), z(2), ..., z(m)], where i is the channel number. The formula for calculating the mean μ is: (2) Calculate the variance, σ 2 : (3) For each channel Perform normalization separately: The above formula transforms the original data distribution into a normal distribution, then The mean becomes 0 and the variance becomes 1, where ∈ takes a small number to prevent the expression from being divided by 0 when the variance is 0. (4) Scale and translate to obtain the translated data. Here, γ and β are training hyperparameters, the purpose of which is to prevent the data from having a linear distribution and to adjust it to appropriate values ​​by continuously learning from the dataset.

6. The BGA solder joint segmentation method based on X-ray imaging according to claim 4, characterized in that: In step S203, the BGA solder joint image is processed by a Butterworth filter to improve the clarity of the image details, and then histogram equalization is used to enhance the contrast of the BGA solder joint image, making the solder joint features more obvious.

7. The BGA solder joint segmentation method based on X-ray imaging according to claim 1, characterized in that: In step S3, an improved U-Net neural network model is used to coarsely segment the BGA solder joint image. The improved U-Net neural network model includes a channel-wise convolution module and a point-wise convolution module. The channel-wise convolution module includes an input channel and a first filter set at the output end of the input channel, which is the same as the number of input channels. The data of one input channel is processed by convolution through the corresponding first filter to obtain an output feature map with the same number of input channels. The pointwise convolution module has several convolution kernels, each with a size of 1×1×M, where M is the number of input channels. Each convolution kernel has a second filter with a size of 1×1×M at its output.

8. The BGA solder joint segmentation method based on X-ray imaging according to claim 1, characterized in that: In step S4, the deep convolutional neural network includes a convolutional pooling module and a fully connected layer module disposed at the output end of the convolutional pooling module. The convolutional pooling module includes several convolutional modules, and each convolutional module has a max pooling layer module at its output end. A ReLU layer is disposed between the convolutional modules and the max pooling layer modules. The output end of the fully connected layer module is the output of the deep convolutional neural network.

9. The BGA solder joint segmentation method based on X-ray imaging according to claim 8, characterized in that: In step S203, the number of layers in the convolutional module during the training of the deep convolutional neural network is 3, the initial number of convolutional channels is 32, and the learning rate is 0.

001.

10. A BGA solder joint segmentation system based on X-ray imaging, used to implement the BGA solder joint segmentation method based on X-ray imaging as described in any one of claims 1-9, characterized in that, include: Image acquisition module: used to acquire raw solder joint images of BGA; Image preprocessing module: used to preprocess the original BGA solder joint imaging image to form a BGA solder joint image; Coarse segmentation module: Used to extract feature information of solder joints from BGA solder joint image, locate solder joints and perform preliminary segmentation, and achieve separation of arbitrary solder joints from the background; Fine segmentation module: It is used to automatically adjust the parameters of the Hough transform circle detection algorithm using a deep convolutional neural network, and then use the automatically adjusted parameter Hough transform circle detection algorithm to learn features from the coarse segmentation results to achieve accurate segmentation of BGA solder joints.

Images