Feature extraction method of chip resistor micro voids and solder joints based on multi-layer convolutional network

Abstract

This invention relates to the technical field of industrial electronic component detection. More specifically, it relates to a method for extracting microvoid and solder joint features from chip resistors based on a multi-layer convolutional network. The key technical aspects of this method are: S1. Constructing training and test datasets for microvoids and solder joints; S2. Detecting microvoids based on a residual multiscale skip connection network (RMSC-Net); S3. Detecting solder joints based on a recurrent convolutional neural network (RU-Net); and S4. Using the trained model to perform detection results. This method offers the advantages of high algorithmic recognition and segmentation accuracy and strong anti-interference capabilities.

Description

Technical Field

[0001] The present invention relates to the technical field of industrial electronic component detection, and more specifically, to a method for extracting micro-voids and solder joint features of chip resistors based on a multi-layer convolutional network. Background Art

[0002] With the rapid development of society, economy, industrial automation, and science and technology, chip resistors, with their advantages of small size, light weight, high mounting density, and strong anti-interference capabilities, are widely used in industrial equipment, medical devices, high-end computers, and medical electronics. They are currently one of the most widely used chip electronic components in the electronics industry. However, improper flux control during the soldering process can cause air bubbles to form voids of varying sizes in the solder joints during the reflow process. The void rate of solder joints is closely related to the electrical and thermal conductivity of the chip resistors, making an accurate and stable algorithm for detecting micro-voids and solder joints in chip resistors particularly important.

[0003] Currently, commonly used methods for extracting solder joint and void features include edge detection, multi-scan algorithms, BPN neural network algorithms, and convolutional neural networks. While these algorithms can meet void and solder joint detection requirements to a certain extent, they also have limitations. For example, edge detection methods cannot accurately identify micro-voids, while multi-scan algorithms are prone to false void detection. Summary of the Invention

[0004] In view of the shortcomings of the existing technology, the purpose of the present invention is to provide a chip resistor micro-void and solder joint feature extraction method based on a multi-layer convolutional network, which has the advantages of high algorithm recognition and segmentation accuracy and strong anti-interference ability.

[0005] The above technical objectives of the present invention are achieved through the following technical solutions: a method for extracting micro voids and solder joint features of chip resistors based on a multi-layer convolutional network, comprising the following steps:

[0006] S1, build micro void and solder joint training dataset and test dataset;

[0007] S2, micro-void detection based on residual multi-scale skip connection network (RMSC-Net);

[0008] S3, solder joint detection based on recurrent convolutional neural network (RU-Net);

[0009] S4. Use the trained model to perform detection and obtain the detection results.

[0010] In one embodiment, step S1 specifically includes the following steps:

[0011] S11. Set the acceleration voltage, target power, and magnification of the CT device to obtain clear images of chip resistor micro-voids and solder joints;

[0012] S12. Collect sample image data using a CT device, and annotate the collected chip resistor microvoids and solder joint data so that each annotated file contains the characteristics and location information of the microvoids and solder joints in the image;

[0013] S13. Improve the generalization of the network by cropping, rotating and changing the brightness to achieve data enhancement;

[0014] S14, inputting the enhanced micro-void and solder joint training data into RMSC-Net and RU-Net respectively for training, wherein RMSC-Net extracts features of micro-voids and RU-Net extracts features of solder joints;

[0015] S15. The weights of neurons are adjusted through the binary cross entropy loss function, and the output results of the two networks are fused to finally realize the feature extraction of micro voids and solder joints.

[0016] In one embodiment, the step S11 is specifically as follows:

[0017] In the 2D X-ray detection mode, the acceleration voltage and target power of the CT device are set to 100 kV and 4 W respectively, and the magnification is set according to the size of the sample, with the magnification being between 10-50 times;

[0018] The image resolution of the CT device is 1004×1004. To make the output image of each network consistent with the input image size, 2 pixels are padded on the top, bottom, left, and right of the 1004×1004 pixel image to obtain a 1008×1008 image.

[0019] Since the edges of the CT image are white, the image is preprocessed and filled with (255, 255, 255).

[0020] In one embodiment, step S2 specifically includes the following steps:

[0021] The S21 and RMSC-Net architectures include convolutional layers, pooling layers, residual layers, multi-scale skip connection layers, and upsampling layers. The X-ray images collected by the CT device and their corresponding micro-void labels, including the data-enhanced images and corresponding labels, are input into the convolutional layers. The convolutional layers used are two with 64 convolution kernels each. Each convolutional layer is a composite function of a convolution operation with a filter size of 3×3 and a padding of 1, batch normalization, and a LeakyReLU activation function. The LeakyReLU activation function is defined as:

[0022] LeakyReLU(x)=max(0,x)+negative_slope*min(0,x)

[0023] After the S22 convolution layer, a maximum pooling layer with a filter size of 2 is used to obtain high-level semantic features of the image and remove redundant information from the X-ray image. The maximum pooling is followed by two convolutional layers with 128 convolution kernels each. The encoder has a total of 4 maximum pooling layers, and the number of feature maps after the first three maximum pooling layers is doubled. To reduce the amount of computation, the number of feature maps after the last pooling layer remains unchanged. The bottleneck stage after the last pooling layer uses residual connections to solve the problem of gradient diffusion or gradient explosion. The residual module is defined as:

[0024] y out =F(x in ,{W i})+x in

[0025] Among them, x in and y out are the input and output vectors of the layer under consideration, respectively, and the function F(x in ,{W I}) represents the residual mapping to be learned.

[0026] In one embodiment, step S3 is specifically as follows:

[0027] RU-Net is used to extract features from solder joints of chip resistors. The input image size is 1008×1008 pixels. Shallow convolution focuses on texture features, while deep networks focus on high-dimensional features. Therefore, RU-Net performs feature concatenation to extract edge features of solder joints.

[0028] The X-ray image and its corresponding solder joint labels are input into RU-Net. The BCEWithLogitsLoss loss function is used for loss calculation and weight update, which is as follows:

[0029] l(x,y)=L={l1,...,l N} T ,l n =-w n [y n glogσ(x n )+(1-y n )glog(1-σ(x n ))]

[0030] Where x is the input, y is the target, w is the weight, N is the batch size, and σ(x) is the Sigmoid function, which can be expressed as:

[0031]

[0032] In order to adapt to the detection of solder joints, the RMSprop optimizer is used and the learning rate is set to 10 -5 .

[0033] In one embodiment, step S4 specifically includes the following steps:

[0034] S41. During the image data acquisition process, different magnifications are selected according to the sample size. The size of the cavity ranges from a few microns to hundreds of microns, so the magnification we choose is between 10-50 times;

[0035] S42. Three chip resistors, model 0402, 0603, and 0805, were used to train and test RMSC-Net and RU-Net using chip resistor images and void and solder joint data. The data from the three models were mixed for training. Different data augmentation methods were used for the two different detection targets, voids and solder joints: the original 45-group void dataset was expanded to 236 groups through random cropping and rotation data augmentation; the original 83-group solder joint dataset was expanded to 150 groups through random rotation and brightness modification data augmentation.

[0036] S43, the network uses Pytorch, Python version 3.7; the hardware conditions are NVIDIA GeForce RTX 3090; the memory is 24GB; in the process of void and solder joint detection, only the network used is different, the training and testing steps and hyperparameters are exactly the same, and the learning rate is set to 10 -5 , using RMSprop optimizer and BCEWithLogitsLoss loss function;

[0037] S44. In order to quantitatively analyze the performance of RMSC-Net and RU-Net networks, several performance indicators are considered, including precision (PR), set similarity (DC), sensitivity (SE), specificity (SP), category average pixel accuracy (MPA) and mean intersection over union (MIOU), where:

[0038]

[0039]

[0040]

[0041]

[0042]

[0043]

[0044] The images output by RMSC-Net and RU-Net are evaluated using the above evaluation formula, including qualified and unqualified results. A solder joint with a void rate exceeding 10% is defined as unqualified and set as Failed; a qualified one is set as Passed.

[0045] The above-mentioned chip resistor micro-void and solder joint feature extraction method based on a multi-layer convolutional network has the following beneficial effects:

[0046] This method first designs a prediction method that combines RMSC-Net and RU-Net, and utilizes the powerful feature expression and learning capabilities of multi-layer convolutional neural networks and recurrent convolutional networks to enhance the network's adaptability and prediction capabilities; introduces a residual layer to ensure the network's prediction accuracy, designs multi-scale jump connections for multi-dimensional feature fusion, and fully utilizes feature information to achieve accurate feature perception of micro-voids and solder joints; obtains micro-void and solder joint detection results with higher accuracy than traditional algorithms and other deep learning networks. This method has high recognition and segmentation accuracy and strong anti-interference ability.

[0047] The present invention redesigns the multi-scale jump connection layer to fuse features from multiple different levels, which helps to restore the information loss caused by the pooling operation and reduces the complexity of the network. The encoder stage starts from the last layer of downsampling, and performs feature fusion on high-level and low-level semantic feature maps through bilinear interpolation and multi-scale jump connection. The first multi-scale feature fusion is to fuse the upsampled features with the features of the third and fourth layers of the encoder to obtain 1280 feature maps. Feature restoration is performed after 4 upsamplings, 1 residual connection and 3 multi-scale jump feature fusions. Finally, a 1×1 convolution operation is used to adjust the channel and output the empty feature prediction results. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 1 is a flowchart of the steps of the extraction method in this embodiment;

[0049] Figure 2 This is a picture of the chip resistor sample in this embodiment;

[0050] Figure 3 This is an imaging diagram of the chip resistor of the CT device in this embodiment;

[0051] Figure 4 is a flow chart of the method for extracting micro voids and solder joint features of chip resistors in this embodiment;

[0052] Figure 5 This is the RMSC-Net network architecture in this embodiment;

[0053] Figure 6 This is a comparison chart of the actual detection results of micro voids and solder joints of the chip resistor in this embodiment and the network prediction results. DETAILED DESCRIPTION

[0054] The present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0055] like Figure 1 As shown in FIG, a chip resistor micro-void and solder joint feature extraction method based on a multi-layer convolutional network includes the following steps:

[0056] S1. Construct micro void and solder joint training dataset and test dataset.

[0057] Specifically, such as Figures 2 to 4 As shown, step S1 specifically includes the following steps:

[0058] S11. Set the acceleration voltage, target power, and magnification of the CT device to obtain clear images of chip resistor micro-voids and solder joints.

[0059] Specifically, step S11 is as follows: in the two-dimensional X-ray detection mode, the acceleration voltage and target power of the CT device are set to 100 kV and 4 W respectively, and the magnification is set according to the size of the sample, and the magnification is between 10 and 50 times; the image resolution collected by the CT device is 1004×1004, and in order to make the output image of each network consistent with the input image size, 2 pixels are padded on the top, bottom, left and right of the 1004×1004 pixel image to obtain a 1008×1008 image; because the edges of the CT image are white, the image is pre-processed and padded with (255, 255, 255).

[0060] Sample image data of chip resistors, i.e. Figure 2 , X-rays are emitted through an X-ray generator, and the light beam after being collimated by a collimator passes through a certain fault of the resistor sample. Depending on the density and atomic weight of the sample material itself, there are different amounts of X-ray absorption. The X-ray receiver is used to sense the intensity of the X-rays passing through each part, and the collected data is reconstructed by a computer. According to the X-ray intensity of each part, it is converted into a black and white picture. The strong intensity is white, and the low intensity is black. Among them, circles, ellipses and irregular polygons such as 01, 02, and 03 represent microvoids, and rectangular boxes such as 04 represent the edges of solder joints. Microvoids have a relatively obvious dividing line. The microvoids inside the dividing line are white, and the microvoids outside are black, that is, Figure 3 .

[0061] S12. Collect sample image data through CT equipment, and annotate the collected chip resistor micro-voids and solder joint data so that each annotated file contains the characteristics and position information of the micro-voids and solder joints in the image.

[0062] S13. Improve the generalization of the network by cropping, rotating and changing the brightness to achieve data enhancement.

[0063] S14. The enhanced training data of micro voids and solder joints are input into RMSC-Net and RU-Net for training respectively. RMSC-Net extracts the features of micro voids, and RU-Net extracts the features of solder joints.

[0064] S15. The weights of neurons are adjusted through the binary cross entropy loss function, and the output results of the two networks are fused to finally realize the feature extraction of micro voids and solder joints.

[0065] S2. Detect micro-voids based on the residual multi-scale skip connection network (RMSC-Net).

[0066] Specifically, such as Figure 5 As shown, step S2 specifically includes the following steps:

[0067] The S21 and RMSC-Net architectures include convolutional layers, pooling layers, residual layers, multi-scale skip connection layers, and upsampling layers. The X-ray images collected by the CT device and their corresponding micro-void labels, including the data-enhanced images and corresponding labels, are input into the convolutional layers. The convolutional layers used are two with 64 convolution kernels each. Each convolutional layer is a composite function of a convolution operation with a filter size of 3×3 and a padding of 1, batch normalization, and a LeakyReLU activation function. The LeakyReLU activation function is defined as:

[0068] LeakyReLU(x)=max(0,x)+negative_slope*min(0,x)

[0069] Batch normalization speeds up the convergence of the loss function. The LeakyReLU activation function solves the problem of neuron death in the ReLU function. For negative input values, backpropagation can also be performed due to the small positive slope in the negative area.

[0070] After the S22 convolution layer, a maximum pooling layer with a filter size of 2 is used to obtain high-level semantic features of the image and remove redundant information from the X-ray image. The maximum pooling is followed by two convolutional layers with 128 convolution kernels each. The encoder has a total of 4 maximum pooling layers, and the number of feature maps after the first three maximum pooling layers is doubled. To reduce the amount of computation, the number of feature maps after the last pooling layer remains unchanged. The bottleneck stage after the last pooling layer uses residual connections to solve the problem of gradient diffusion or gradient explosion. The residual module is defined as:

[0071] y out =F(xin ,{W i})+x in

[0072] Among them, x in and y out are the input and output vectors of the layer under consideration, respectively, and the function F(x in ,{W I}) represents the residual mapping to be learned.

[0073] The present invention redesigns the multi-scale jump connection layer to fuse features from multiple different levels, which helps to restore the information loss caused by the pooling operation and reduces the complexity of the network. The encoder stage starts from the last layer of downsampling and performs feature fusion on high-level and low-level semantic feature maps through bilinear interpolation and multi-scale jump connection. The first multi-scale feature fusion is to fuse the upsampled features with the features of the third and fourth layers of the encoder to obtain 1280 feature maps. Feature restoration is performed after 4 upsamplings, 1 residual connection and 3 multi-scale jump feature fusions. Finally, a 1×1 convolution operation is used for channel adjustment to output the empty feature prediction results.

[0074] S3. Detect solder joints based on recurrent convolutional neural network (RU-Net).

[0075] Specifically, step S3 is as follows: RU-Net is used to extract the features of the solder joints of the chip resistors. The image size of the input network is 1008×1008 pixels. The shallow convolution focuses on texture features, while the deep network focuses on high-dimensional features. Therefore, feature splicing is performed through RU-Net to realize the edge feature extraction of the solder joints.

[0076] The X-ray image and its corresponding solder joint labels are input into RU-Net. The BCEWithLogitsLoss loss function is used for loss calculation and weight update, which is as follows:

[0077] l(x,y)=L={l1,...,l N} T ,l n =-w n [y n glogσ(x n )+(1-y n )glog(1-σ(x n ))]

[0078] Where x is the input, y is the target, w is the weight, N is the batch size, and σ(x) is the Sigmoid function, which can be expressed as:

[0079]

[0080] In order to adapt to the detection of solder joints, the RMSprop optimizer is used and the learning rate is set to 10 -5 .

[0081] S4. Use the trained model to perform detection and obtain the detection results.

[0082] Specifically, step S4 includes the following steps:

[0083] S41. During the image data acquisition process, different magnifications are selected according to the sample size. The size of the cavity ranges from a few microns to hundreds of microns, so the magnification we choose is between 10-50 times;

[0084] S42. Three chip resistors, model 0402, 0603, and 0805, were used to train and test RMSC-Net and RU-Net using chip resistor images and void and solder joint data. The data from the three models were mixed for training. Different data augmentation methods were used for the two different detection targets, voids and solder joints: the original 45-group void dataset was expanded to 236 groups through random cropping and rotation data augmentation; the original 83-group solder joint dataset was expanded to 150 groups through random rotation and brightness modification data augmentation.

[0085] S43, the network uses Pytorch, Python version 3.7; the hardware conditions are NVIDIA GeForce RTX 3090; the memory is 24GB; in the process of void and solder joint detection, only the network used is different, the training and testing steps and hyperparameters are exactly the same, and the learning rate is set to 10 -5 , using RMSprop optimizer and BCEWithLogitsLoss loss function;

[0086] S44. In order to quantitatively analyze the performance of RMSC-Net and RU-Net networks, several performance indicators are considered, including precision (PR), set similarity (DC), sensitivity (SE), specificity (SP), category average pixel accuracy (MPA) and mean intersection over union (MIOU), where:

[0087]

[0088]

[0089]

[0090]

[0091]

[0092]

[0093] The images output by RMSC-Net and RU-Net are evaluated using the above evaluation formula, including qualified and unqualified results. A solder joint with a void rate exceeding 10% is defined as unqualified and set as Failed; a qualified one is set as Passed.

[0094] The images output by RMSC-Net were evaluated using the evaluation formula. The corresponding evaluation metrics for the 196 microvoid images were 71.07%, 78.38%, 87.50%, 99.35%, 93.43%, and 81.86%, respectively. The images output by RU-Net were evaluated using the evaluation formula. The corresponding evaluation metrics for the 158 solder joint images were 96.95%, 98.36%, 99.86%, 98.88%, 99.37%, and 97.82%, respectively. The evaluation metrics are described in the order in which the formulas are used.

[0095] like Figure 6 As shown in the figure, the actual results are compared with the predicted results output by RMSC-Net and RU-Net. The first row is the original image of the micro-void and solder joint dataset, the second row is the actual detection results of micro-void and solder joint, and the last row is the network prediction result image. The models corresponding to the images are (a) 0402, (b) 0603, and (c) 0805. The micro-void and solder joint feature result images are binarized, the rectangular box represents the edge of the solder joint, and the circle, ellipse, and irregular polygon represent the information features of the micro-void. It can be seen from the figure that the detection results of the method proposed in the present invention are very close to the actual results, which proves the accuracy and effectiveness of the proposed method.

[0096] Compared with the existing technology, the beneficial effects and advantages of the present invention are as follows: The present invention designs a method for extracting micro-voids and solder joint features of chip resistors based on a multi-layer convolutional network. The powerful feature expression and learning capabilities of multi-layer convolutional neural networks and recurrent convolutional networks are utilized to enhance the network's adaptability and prediction capabilities. This method first designs a prediction method that combines RMSC-Net and RU-Net; introduces a residual layer to ensure the prediction accuracy of the network, designs a multi-scale jump connection for multi-dimensional feature fusion, and fully utilizes feature information to achieve accurate feature perception of micro-voids and solder joints; compared with the micro-void and solder joint detection results of traditional algorithms and other deep learning networks, this method has higher recognition and segmentation accuracy and stronger anti-interference ability.

[0097] The above-described embodiments merely illustrate several implementations of the present invention, and while their descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art would be able to make numerous variations and improvements without departing from the spirit of the present invention, all of which fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be determined by the appended claims.

Claims

1. A chip resistor micro-void and solder joint feature extraction method based on a multi-layer convolutional network, characterized by: The following steps are involved: S1. Set the acceleration voltage, target power, and magnification of the CT device to obtain clear images of chip resistor micro-voids and solder joints; S2. Collect sample image data using CT equipment and annotate the collected chip resistor micro-voids and solder joint data so that each annotated file contains the characteristics and location information of the micro-voids and solder joints in the image; S3, improve the generalization of the network by cropping, rotating and changing the brightness to achieve the effect of data enhancement; S4. The enhanced training data of micro voids and solder joints are input into RMSC-Net and RU-Net for training respectively. RMSC-Net extracts the features of micro voids, and RU-Net extracts the features of solder joints. S5. Adjust the weights of neurons through the binary cross entropy loss function, fuse the outputs of the two networks, and finally achieve feature extraction of micro voids and solder joints; S6. Use the trained model to perform detection and obtain the detection results; The step S1 is specifically as follows: In the 2D X-ray detection mode, the acceleration voltage and target power of the CT device are set to 100 kV and 4 W respectively, and the magnification is set according to the size of the sample, with the magnification being between 10-50 times; The image resolution of the CT device is 1004×1004. To make the output image of each network consistent with the input image size, 2 pixels are padded on the top, bottom, left, and right of the 1004×1004 pixel image to obtain a 1008×1008 image. Since the edges of the CT image are white, the image is preprocessed and filled with (255, 255, 255); The step S4 specifically includes the following steps: The S41 and RMSC-Net architectures include convolutional layers, pooling layers, residual layers, multi-scale skip connection layers, and upsampling layers. The X-ray images collected by the CT device and their corresponding micro-void labels, including the data-enhanced images and corresponding labels, are input into the convolutional layers. The convolutional layers used are two with 64 convolution kernels each. Each convolutional layer is a composite function of a convolution operation with a filter size of 3×3 and a padding of 1, batch normalization, and a LeakyReLU activation function. The LeakyReLU activation function is defined as: LeakyReLU(x)=max(0,x)+negative_slope*min(0,x) After the S42 convolution layer, a maximum pooling layer with a filter size of 2 is used to obtain high-level semantic features of the image and remove redundant information from the X-ray image. The maximum pooling is followed by two convolutional layers with 128 convolution kernels each. The encoder has a total of 4 maximum pooling layers, and the number of feature maps after the first three maximum pooling layers is doubled. To reduce the amount of computation, the number of feature maps after the last pooling layer remains unchanged. The bottleneck stage after the last pooling layer uses residual connections to solve the problem of gradient diffusion or gradient explosion. The residual module is defined as: y out =F(x in ,{W i })+x in Among them, x in and y out are the input and output vectors of the layer under consideration, and the function F(x in ,{W i }) represents the residual mapping to be learned.

2. The chip resistor micro-void and solder joint feature extraction method based on a multi-layer convolutional network according to claim 1 is characterized in that: The step S4 is specifically as follows: RU-Net is used to extract features from solder joints of chip resistors. The input image size is 1008×1008 pixels. Shallow convolution focuses on texture features, while deep networks focus on high-dimensional features. Therefore, RU-Net performs feature concatenation to extract edge features of solder joints. The X-ray image and its corresponding solder joint labels are input into RU-Net. The BCEWithLogitsLoss loss function is used for loss calculation and weight update, which is as follows: l(x,y)=L={l1,...,l N } T ,l n =-w n [y n ·logσ(x n )+(1-y n )·log(1-σ(x n ))] Where x is the input, y is the target, w is the weight, N is the batch size, and σ(x) is the Sigmoid function, which can be expressed as: In order to adapt to the detection of solder joints, the RMSprop optimizer is used and the learning rate is set to 10 -5 .

3. The chip resistor micro-void and solder joint feature extraction method based on a multi-layer convolutional network according to claim 1 is characterized in that: The step S6 specifically includes the following steps: S61. During the image data acquisition process, the magnification is selected to be between 10 and 50 times; S62. Three chip resistors, model 0402, 0603, and 0805, were used to train and test RMSC-Net and RU-Net using chip resistor images and void and solder joint data. The data from the three models were mixed for training. For the two different detection targets of voids and solder joints, the original 45-group void dataset was expanded to 236 groups through random cropping and rotation data augmentation. The original 83-group solder joint dataset was expanded to 150 groups through random rotation and brightness modification data augmentation. S63, the network uses Pytorch, Python version 3.7; the hardware conditions are NVIDIA GeForce RTX 3090; the memory is 24GB; in the process of void and solder joint detection, only the network used is different, the training and testing steps and hyperparameters are exactly the same, and the learning rate is set to 10 -5 , using RMSprop optimizer and BCEWithLogitsLoss loss function; S64. In order to quantitatively analyze the performance of RMSC-Net and RU-Net networks, several performance indicators are considered, including precision (PR), set similarity (DC), sensitivity (SE), specificity (SP), category average pixel accuracy (MPA) and mean intersection over union (MIOU), where: The images output by RMSC-Net and RU-Net are evaluated using the above evaluation formula.

Images