Chip weld cavity defect detection method and system

Abstract

The present application relates to chip defect detection technical field, disclose a kind of chip weld cavity defect detection method and system, the method, comprising the following steps: S1, weld extraction: chip weld is extracted from original chip image, generates the image containing weld area;S2, image block: original chip image is decomposed into several overlapping image blocks, and combined with the extracted weld area, the image block containing weld area is screened out;S3, cavity segmentation: the cavity in the screened image block is segmented;S4, image restoration: based on the reconstructed image block after segmentation, generate the mask chart same as original chip image size;S5, decision: the minimum welding width of chip is calculated using the generated image containing weld area, mask chart, and whether the chip is qualified according to minimum welding width is judged.The present application solves the problems of existing technology, such as easy overfitting of model, poor cavity segmentation effect and the like.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of chip defect detection, and particularly relates to a chip weld seam cavity defect detection method and system. BACKGROUND

[0002] As an important process in chip production, alloy sealing is widely used in many microwave semiconductor devices, integrated circuits and MEMS applications that require high airtightness. During the sealing process, factors such as solder composition, surface quality and process can cause bubbles to form inside the weld seam, thereby affecting the quality of the chip. With the rapid development of the integrated circuit industry, users' requirements for chip quality are also increasing day by day. In order to improve the production yield of chips, detecting chip weld seam defects during chip production has become an indispensable part of intelligent manufacturing. Usually, such bubbles are difficult to observe directly by the human eye, and often need to use X-rays to scan the weld seam, so that the bubbles appear as cavities in the image to facilitate the inspector to conduct inspection. At present, enterprises mainly use manual visual inspection method for chip cavity defect detection. Manual visual inspection is simple and convenient, but has low detection efficiency, poor accuracy, high work intensity of workers, and is prone to cause visual fatigue, which cannot meet the growing production demand.

[0003] In recent years, deep learning-based defect detection models have shown great advantages in reducing human errors. With the development of deep learning, it has been widely used in chip or printed circuit board defect detection due to its excellent performance. Zheng et al. proposed a hybrid algorithm for defect detection of light-emitting diode chips. The algorithm first uses a geometric calculation method to perform coarse defect detection on large-scale chip lithography images in the form of grid segmentation; then, the algorithm uses a convolutional neural network model to perform secondary fine detection on the "suspected defect" chips checked in the above step. Stern et al. proposed a fully convolutional network for defect classification of light-emitting diode chips. The network consists of three parts: a CNN part that extracts semantic information from the input image, an upsampling part that restores spatial information, and a skip connection part that improves resolution. Wang et al. proposed a target detection network, YOLOV4-SA, for detecting small defects on the surface of chips. The network consists of a spatial attention module, a feature extraction backbone, and a feature fusion module. This method can correct the values of the feature map and highlight the defect area, making it more effective in identifying small defects. Park et al. proposed a prediction model based on Bayesian neural networks. The model uses the relative frequency of each class and the prediction uncertainty at the sample and class levels to flexibly adjust the decision boundary, effectively predicting chip defects during the packaging test process. Wang et al. used an extremely deep super-resolution algorithm to reconstruct high-resolution acoustic microscope images, and then designed a classification model based on cnn to classify the solder joints in flip-chip images. Ling et al. proposed a new deep twin semantic segmentation network for PCB solder defect detection. The network consists of two weight-shared encoders, a decoder, and some related modules. In the network, the decoder combines the deep features extracted from the two encoders with their corresponding feature difference maps to restore spatial information. The feature difference map is calculated through the related module to make the network pay more attention to semantic differences and further alleviate the overfitting problem. Finally, the network uses a combined loss function to further improve the performance of small defects.

[0004] However, the scarcity of defect data makes deep learning models prone to overfitting during training. At the same time, the large gray difference between chips and the uneven gray distribution on the hollow surface also pose great challenges to the algorithm.

[0005] In summary, the current deep learning-based image segmentation algorithm has the following problems:

[0006] 1) The model is prone to overfitting. In actual production, it is difficult for enterprises to collect original defect chip data. Therefore, the collected chip defect images are usually small, which can easily lead to overfitting of deep learning-based image segmentation models during training.

[0007] 2) The cavity segmentation effect is poor. The chip has a multi-layer structure, and the image background interference is generated after each layer is superimposed after x-ray scanning. The Compton effect generated when the x-ray passes through the chip often leads to uneven image gray scale distribution. In addition, different chip structures have large differences in gray scale values, making it difficult for standard segmentation algorithms to achieve accurate cavity segmentation. SUMMARY

[0008] To overcome the shortcomings of the prior art, the present application provides a chip weld cavity defect detection method and system, which solves the problems of easy overfitting of the model and poor cavity segmentation effect in the prior art.

[0009] The technical scheme adopted by the present application to solve the above problems is:

[0010] A chip weld cavity defect detection method, comprising the following steps:

[0011] S1, weld extraction: extracting the chip weld from the original chip image to generate an image containing the weld area;

[0012] S2, image blocking: decomposing the original chip image into a plurality of overlapping image blocks, and combining the extracted weld area to screen out image blocks containing the weld area;

[0013] S3, cavity segmentation: segmenting the cavities in the screened image blocks;

[0014] S4, image restoration: reconstructing based on the segmented image blocks to generate a mask image with the same size as the original chip image;

[0015] S5, decision: calculating the minimum weld width of the chip using the generated image containing the weld area and the mask image, and determining whether the chip is qualified according to the minimum weld width.

[0016] As a preferred technical scheme, in step S3, the attention cavity segmentation network is used to segment the cavities in each image block.

[0017] As a preferred technical scheme, the attention cavity segmentation network includes one ATT, A+B+1 MFEs, A MFEs are connected to the input end of the ATT, B MFEs are connected to the output end of the ATT, and the A MFEs, 1 MFE and B MFEs are connected in series; wherein, ATT refers to a channel attention unit, MFE refers to a multi-branch feature extraction unit, A and B are integers and both >1.

[0018] As a preferred technical solution, the MFE comprises a first branch, a second branch and a third branch, and the MFE is configured to: given an input feature A, use the first branch, the second branch and the third branch to extract feature maps respectively, and then perform element-by-element summation operation on the feature maps extracted by the first branch, the second branch and the third branch to obtain a final output feature A';

[0019] The first branch comprises a convolution kernel with a kernel size of 3 and a stride of 1; the second branch comprises the following convolution kernels connected in series: a convolution kernel with a kernel size of 3 and a stride of 3, and a convolution kernel with a kernel size of 1 and a stride of 1; and the third branch comprises the following convolution kernels connected in series: a convolution kernel with a kernel size of 3 and a stride of 1, a convolution kernel with a kernel size of 3 and a stride of 3, and a convolution kernel with a kernel size of 1 and a stride of 1.

[0020] As a preferred technical solution, the ATT performs the following steps when working:

[0021] A1, for a plurality of input feature maps of different sizes, adjusting the sizes of the feature maps to the same size;

[0022] A2, splicing the adjusted feature maps in the channel dimension to obtain a new feature map F;

[0023] A3, calculating a channel attention map X of the new feature map F;

[0024] A4, performing matrix multiplication between the transpose of F and the channel attention map X to obtain a weighted feature mapping F';

[0025] A5, finally dividing F' into a plurality of sub-feature maps, and adjusting the sizes of the plurality of sub-feature maps to obtain a plurality of feature maps with the same size as the corresponding input feature maps.

[0026] As a preferred technical solution, in step S5, the minimum welding width of each chip is calculated based on the welding seam area segmented by the SAM and the mask map reconstructed by using a minimum welding width search method with a step decrement.

[0027] As a preferred technical solution, in step S5, the method for judging whether the chip is qualified is: when the minimum welding width is less than r times the design sealing width, the chip is determined to be unqualified; wherein r represents a set coefficient.

[0028] As a preferred technical solution, in step S5, the calculation method of the minimum welding width is:

[0029] B1, constructing an adjacency matrix M of a plurality of holes and two boundaries of a welding seam, any element M[i][j] in the adjacency matrix M representing the distance of hole i and hole j,

[0030]

[0031] In the formula, W is the designed sealing width of the weld, d ij is the nearest distance between the boundary of the hole i and the boundary of the hole j, and when d ij is less than r times the designed sealing width of the weld, it is considered that the two holes can be directly reached, otherwise it is considered that the two holes cannot be directly reached;

[0032] B2, after obtaining the adjacency matrix M between all the holes, a 1×n list U is newly created by using a step-decreasing minimum welding width search method, which is used to record the parent node of each hole, and the initial state sets all the values to -1, indicating that there is no parent node; and a corresponding state list C is established, which is used to record whether the hole has been accessed, and the initial state sets the starting point to True and the rest to False;

[0033] B3, taking a side of the weld as the starting point, selecting the hole with a False state and being able to reach the starting point in C; then, setting the hole as the current vertex and setting its state to True; then, searching whether there is another hole within the set step range of the boundary of the hole: if there is no hole within the set step range, it is considered that the hole cannot reach the final boundary, and the distance of the hole to the starting boundary is set to ∞; if there is another hole within the set step range, the distance of the searched other hole to the starting boundary is updated, and the parent node of the searched other hole is set to the current vertex; then, the searched other hole is set as the current vertex, and the searching is continuously repeated until the state of the end point C[end] is True;

[0034] B4, obtaining the shortest path distance D[end] of the end point to the starting point, if D[end] is ∞, it is considered that the actual welding surface width is greater than r times the designed sealing width, and the chip is qualified; otherwise, the chip is unqualified.

[0035] As a preferred technical solution, in step S1, the weld extraction module extracts the chip weld by using the SAM model.

[0036] A chip weld hole defect detection system is used to implement the chip weld hole defect detection method, and comprises the following modules connected in sequence:

[0037] A weld extraction module is used to extract the chip weld from the original chip image to generate an image containing the weld area;

[0038] An image blocking module is used to decompose the original chip image into a plurality of overlapping image blocks, and screen out the image blocks containing the weld area in combination with the extracted weld area;

[0039] The cavity segmentation module is used to segment the cavity in the screened image block;

[0040] The image restoration module is used to reconstruct the segmented image block to generate a mask image with the same size as the original chip image;

[0041] The decision module is used to calculate the minimum welding width of the chip by using the generated image containing the welding area and the mask image, and to judge whether the chip is qualified according to the minimum welding width.

[0042] Compared with the prior art, the present application has the following beneficial effects:

[0043] (1) The present application uses the SAM model to realize interactive segmentation of the chip welding area without any training;

[0044] (2) The present application constructs a large-scale cavity data set by overlapping block of the welding area to overcome the problem of overfitting of the deep learning algorithm in the case of scarce data;

[0045] (3) The present application proposes an attention cavity segmentation network to segment the cavity existing in the welding seam, and proposes a step-decreasing minimum welding width search method to help users judge whether the chip is qualified. BRIEF DESCRIPTION OF DRAWINGS

[0046] Figure 1 The structure diagram of the chip welding cavity defect detection system is shown in the present application;

[0047] Figure 2 The local enlarged view of Figure 1 ;

[0048] Figure 3 The local enlarged view of Figure 1 ;

[0049] Figure 4 The structure diagram of the multi-branch feature extraction unit is shown in the present application;

[0050] Figure 5 The structure diagram of the channel attention unit is shown in the present application;

[0051] Figure 6 The welding surface width determination diagram is shown in the present application;

[0052] Figure 7 The step-decreasing minimum welding width search method diagram state one is shown in the present application;

[0053] Figure 8 The step-decreasing minimum welding width search method diagram state two is shown in the present application;

[0054] Figure 9Fig. 3 is a schematic diagram of a step-decreasing minimum welding width search method. DETAILED DESCRIPTION

[0055] The present application will be further described below in conjunction with embodiments and drawings, but the embodiments of the present application are not limited thereto.

[0056] Embodiment 1

[0057] As shown in Figures 1 to 9 , in order to solve the problems that existing image segmentation models are prone to overfitting and have poor segmentation effect on chip images with insufficient data, a chip weld hollow defect detection method and system are proposed, which is an interactive chip weld hollow defect detection system.

[0058] A chip weld hollow defect detection system is composed of five cascaded sub-modules: a weld extraction module, an image blocking module, a hollow segmentation module, an image restoration module, and a decision module, and the overall architecture is as shown in Figure 1 .

[0059] More specifically, as follows:

[0060] The weld extraction module (as shown in Figure 2 ) adopts a Segment Anything Model (SAM) model to interactively extract chip welds by manually clicking or drawing a box. The image blocking module decomposes the chip image into a plurality of overlapping image blocks to enable the subsequent hollow segmentation module to identify the characteristics of the micro-hollows and improve the segmentation capability of the model. At the same time, the image blocking method can also construct a large hollow data set to reduce the risk of overfitting in the training process of the hollow segmentation model. The hollow segmentation module uses an attention hollow segmentation network to segment the hollows in each image block. The image restoration module reconstructs the segmented results to generate a mask image with the same size as the original image. The decision module uses a step-decreasing minimum welding width search method to calculate the minimum welding width of each chip based on the SAM segmented weld area and the reconstructed mask image, and determines whether the chip is qualified according to the width.

[0061] Hollow segmentation module:

[0062] Based on the Unet network, an attention hollow segmentation network is proposed to improve the segmentation capability of the hollows, and the network structure is as shown in Figure 3 . In the encoder and decoder, the attention hollow segmentation network embeds a multi-branch feature extraction unit to systematically aggregate multi-scale context information without losing resolution, thereby improving the learning ability of the model. In the middle of the encoder and decoder, the attention hollow segmentation network embeds a channel attention unit to emphasize the interdependent channel feature maps and enhance the feature representation of specific semantics.

[0063] Multi-branch Feature Extraction: Dilated convolution can aggregate multi-scale context information without loss of resolution, which helps to improve the accuracy of semantic segmentation task. In addition, ResNeXt proves that increasing the base is more effective than increasing the depth or width of the network. This multi-branch architecture not only can reduce the network parameters, but also can improve the performance of the network. Based on these ideas, the attention dilated segmentation network constructs a multi-branch feature extraction unit (MFE) and embeds it into the encoding-decoding process of the dilated segmentation network. The structure of the multi-branch feature extraction unit is shown in Figure 4

[0064] Channel Attention Unit: The architecture of the channel attention unit (ATT) is shown in Figure 5 Figure 5 In the formula, resize represents the scaling of the feature map, reshape represents the deformation of the feature map, transpose represents the transposition of the feature map, CxHxW represents the size of the feature map, and CxC represents the size of the channel attention map. Given four input features C1, C2, C3 and C4 of different sizes, first adjust their sizes to the same size, then concatenate the adjusted feature maps in the channel dimension to obtain a new feature map F. Then, calculate the channel attention map X of F. Then, perform matrix multiplication between the transpose of F and the channel attention map X to obtain the weighted feature mapping F'. Finally, divide F' into four parts and adjust their sizes to obtain four feature maps C1', C2', C3' and C4' of the same size as the corresponding input.

[0065] Decision Module:

[0066] The decision module proposes a step-decreasing minimum welding width search method based on the weld area extracted by SAM and the reconstructed mask map to determine whether the final chip is qualified. The specific determination principle is shown in Figure 6

[0067] Take the void detection shown in Figures 7 to 9 as an example, before calculating the minimum welding width, first construct the adjacency matrix M of the three voids and the two boundaries of the weld. Any element M[i][j] in the adjacency matrix M represents the distance between void i and void j. ​​​

[0068]

[0069] In the formula, W is the designed sealing width of the weld, d ij is the nearest distance between the boundary of the hole i and the boundary of the hole j. When d ij is less than r times the designed sealing width of the weld, it is considered that the two points can be directly reached, otherwise it is considered that they cannot be directly reached.

[0070] After obtaining the adjacency matrix M between all the holes, the step-decreasing minimum weld width search method newly creates a 1×n list U for recording the parent node of each node (hole), and sets all the values to "-1" in the initial state, indicating that there is no parent node; a corresponding state list C is established for recording whether the node has been accessed (whether the hole has been accessed), and the initial point is set to "True" in the initial state, and the rest are "False"; a list D of the same size is established for recording the shortest path length from the point to the starting node, and the initial value stores the distance from all points to the starting point; and an empty list P is established to save the shortest path from each hole to the starting point.

[0071] A certain side of the weld area is taken as the starting vertex (starting point), and a hole (such as hole 1 in the figure) whose state is "False" and which can reach the starting point is selected in C. The hole is set as the current vertex, and its state C[1] is set to "True" (indicating that it has been accessed). Then, as shown in the figure, it is searched whether there is another hole in the range of r×W-D[1] around the boundary of hole 1. If there is no hole in the range, it is considered that the node cannot reach the final boundary, and the corresponding D[·] is set to ∞. If there is (i.e., hole 2 in the figure satisfies d 12 <r×W-D[1]), the distance D[2] of node 2 to the starting boundary is updated to D[1]+d 12 , and the parent node U[2] of node 2 is set to 1. Similarly, as shown in the figure, node 2 is taken as the current node, and it is searched whether there is another hole in the range of r×W-D[2] around it. The above operation is repeated until the state of the end node C[end] is "True" (indicating that it has not been accessed). Figure 7 Figure 8 Finally, the shortest path distance D[end] from the end point to the starting point is obtained, and if D[end] is ∞, it is considered that the actual weld surface width is greater than r times the designed sealing width, and the chip is qualified; otherwise, the chip is unqualified (as shown in the figure).

[0072] Figure 9

[0073] The present application has the following characteristics:

[0074] ​​​1) Attention hole segmentation network. In the encoder and decoder, the multi-branch feature extraction unit is embedded to systematically aggregate multi-scale context information without losing resolution, improving the model learning ability. In the middle of the encoder and decoder, the attention hole segmentation network embeds a channel attention unit to emphasize mutually dependent channel feature maps and enhance the feature representation of specific semantics.

[0075] 2) Step-decreasing minimum weld width search method. First, the adjacency matrix of each hole and the weld edge is calculated, then the step-decreasing method is used to search the minimum weld width of the weld according to the matrix, and whether the chip is qualified is determined.

[0076] The present application has the following beneficial effects:

[0077] (1) The present application uses the SAM model to realize interactive segmentation of the chip weld area without any training;

[0078] (2) The present application constructs a large-scale hole data set by overlapping and blocking the weld area to overcome the problem of overfitting of deep learning algorithms in the case of scarce data;

[0079] (3) The present application proposes an attention hole segmentation network to segment the holes existing in the weld, and proposes a step-decreasing minimum weld width search method to help users determine whether the chip is qualified.

[0080] As described above, the present application can be better implemented.

[0081] All features disclosed in all examples in the specification, or all steps in the methods or processes impliedly disclosed, can be combined and / or extended, replaced, unless mutually exclusive features and / or steps are mutually exclusive.

[0082] The above is only a preferred embodiment of the present application, and does not limit the present application in any form. According to the technical essence of the present application, any simple modification, equivalent replacement and improvement of the above embodiment within the spirit and principles of the present application are still within the protection scope of the technical solution of the present application.

Claims

1. A method for detecting void defects in chip solder joints, characterized in that, Includes the following steps: S1, Weld seam extraction: Extract the chip weld seam from the original chip image and generate an image containing the weld seam area; S2, Image Blocking: Decompose the original chip image into several overlapping image blocks, and combine them with the extracted weld seam area to filter out the image blocks containing the weld seam area; S3, Hole Segmentation: Segmenting holes in the selected image patch; S4, Image Restoration: Reconstruct the image based on the segmented image blocks to generate a mask image of the same size as the original chip image; S5, Decision: Calculate the minimum solder width of the chip using the generated image and mask image containing the solder seam area, and determine whether the chip is qualified based on the minimum solder width; In step S5, the method for calculating the minimum weld width is as follows: B1. Construct an adjacency matrix M between multiple voids and the two boundaries of the weld. Any element M[i][j] in the adjacency matrix M represents the distance to void i and the distance to void j. ； In the formula, W is the designed sealing width of the weld, and dij is the closest distance between the boundary of cavity i and the boundary of cavity j. When dij is less than r times the designed sealing width of the weld, it is considered that the two cavities can be directly reached; otherwise, it is considered that they cannot be directly reached. B2. After obtaining the adjacency matrix M between all holes, a new 1×n list U is created using the minimum weld width search method with decreasing step size to record the parent node of each hole. In the initial state, all values ​​are set to -1, indicating that there is no parent node. A corresponding state list C is also created to record whether a hole has been visited. In the initial state, the starting point is set to True, and the rest are False. B3. Starting from one side of the weld, select a void in C that is in the state of False and reachable from the starting point. Then, set this void as the current vertex and set its state to True. Next, search for other voids within a set step size range of the void's boundary: if no void exists within the set step size range, it is considered that the void cannot reach the final boundary, and the distance from the void to the starting boundary is set to ∞; if a void exists within the set step size range, update the distance from the other voids to the starting boundary, and set the parent node of the other voids as the current vertex; then set the other voids as the current vertex again and continue searching until the state of the final node C[end] is True; where ∞ represents infinity. B4, obtain the shortest path distance D[end] from the end point to the start point. If D[end] is ∞, then the chip is considered to be qualified if the actual welding surface width is greater than r times the designed sealing width; otherwise, the chip is unqualified.

2. The method for detecting chip solder joint void defects according to claim 1, characterized in that, In step S3, the attention-based hole segmentation network is used to segment the holes in each image block.

3. The method for detecting chip solder joint void defects according to claim 2, characterized in that, The attention-based segmentation network consists of one ATT and A+B+1 MFEs. A MFEs are connected to the input of the ATT, and B MFEs are connected to the output of the ATT. A MFEs, 1 MFE, and B MFEs are connected in series. Here, ATT refers to the channel attention unit, MFE refers to the multi-branch feature extraction unit, and A and B are both integers and both > 1.

4. The method for detecting chip solder joint void defects according to claim 3, characterized in that, MFE includes a first branch, a second branch, and a third branch. MFE is used to: given an input feature A, extract feature maps using the first branch, the second branch, and the third branch respectively, and then sum the feature maps extracted by the first branch, the second branch, and the third branch element by element to obtain the final output feature A'. The first branch includes a convolutional kernel with a kernel size of 3 and a spacing of 1; the second branch includes the following convolutional kernels connected in series: a convolutional kernel with a kernel size of 3 and a spacing of 3, and a convolutional kernel with a kernel size of 1 and a spacing of 1; the third branch includes the following convolutional kernels connected in series: a convolutional kernel with a kernel size of 3 and a spacing of 1, a convolutional kernel with a kernel size of 3 and a spacing of 3, and a convolutional kernel with a kernel size of 1 and a spacing of 1.

5. The method for detecting chip solder joint void defects according to claim 3, characterized in that, When ATT is working, perform the following steps: A1: For multiple input feature maps of different scales, adjust their scales to the same size; A2, the adjusted feature maps are concatenated along the channel dimension to obtain a new feature map F; A3, Calculate the channel attention map X of the new feature map F; A4, perform matrix multiplication between the transpose of F and the channel attention map X to obtain the weighted feature map F'; A5, finally divide F' into multiple sub-feature maps, and readjust the size of multiple sub-feature maps to obtain multiple feature maps with the same size as the corresponding input feature map.

6. The method for detecting chip solder joint void defects according to claim 1, characterized in that, In step S5, the minimum solder width search method with decreasing step size is used to calculate the minimum solder width of each chip based on the solder seam area segmented by SAM and the reconstructed mask image.

7. The method for detecting chip solder joint void defects according to claim 1, characterized in that, In step S5, the method for determining whether a chip is qualified is as follows: when the minimum solder width is less than r times the designed sealing width, the chip is determined to be unqualified; where r represents a set coefficient.

8. A method for detecting chip solder joint void defects according to any one of claims 1 to 7, characterized in that, In step S1, the weld seam extraction module uses the SAM model to extract the chip weld seams.

9. A chip solder joint void defect detection system, characterized in that, A method for detecting chip solder joint void defects according to any one of claims 1 to 8 includes the following modules connected in sequence: Weld seam extraction module: used to extract chip weld seams from the original chip image and generate an image containing the weld seam area; Image segmentation module: used to decompose the original chip image into several overlapping image blocks, and combined with the extracted weld seam area, to filter out the image blocks containing the weld seam area; Hole segmentation module: Used to segment holes in the selected image patch; Image restoration module: used to reconstruct the image based on the segmented image blocks and generate a mask image of the same size as the original chip image; Decision module: Used to calculate the minimum solder width of the chip using the generated image and mask image containing the solder seam area, and to determine whether the chip is qualified based on the minimum solder width.

Images