Method for Extracting Solder Joint Areas of Semiconductor Chips Based on a Multi-Layer Atrous Convolution Network

Abstract

The present invention provides a method for extracting solder joint areas of semiconductor chips based on a multi-layer dilated convolutional network. The method includes randomly and evenly dividing the original semiconductor chip solder joint image dataset into N batches; annotating the first batch of the semiconductor chip solder joint image dataset; training the constructed multi-layer dilated convolutional network model using the annotated first batch of the semiconductor chip solder joint image dataset and generating the optimal model at the current stage; inferring the unannotated semiconductor chip solder joint image dataset of the next batch; until the Nth batch of the semiconductor chip solder joint image dataset is processed, performing final model training using all the annotated semiconductor solder joint image datasets to obtain the optimal extraction model for extracting the semiconductor chip solder joint areas. This method can significantly improve the network performance while reducing the number of model parameters, ensure high-quality dataset annotation, and effectively improve the efficiency and accuracy of semiconductor chip solder joint area extraction.

Description

Technical Field

[0001] The present invention relates to a method for extracting solder joint areas of semiconductor chips based on a multi-layer dilated convolutional network, belonging to the technical field of computer vision. Background Art

[0002] With the rapid development of China's electronic information industry, semiconductor chips are increasingly widely used. How to manufacture semiconductor chips more efficiently and at lower cost has become a difficult problem faced by more and more enterprises, and the accurate identification and positioning of the solder joint areas of semiconductor chips are particularly important in the whole production. Therefore, the method for extracting solder joint areas of semiconductor chips is of great significance for improving chip quality.

[0003] At present, most enterprises still identify the solder joint areas of semiconductor chips manually by visual inspection. Manual detection has the disadvantages of low efficiency, high cost, and untimely feedback. Therefore, using intelligent algorithms to design an efficient and automated semiconductor chip solder joint recognition algorithm has become a research hotspot in this field.

[0004] In recent years, semantic segmentation algorithms based on deep learning have received extensive attention from experts and scholars. With the continuous development of convolutional neural networks, some classic semantic segmentation methods based on convolutional neural networks have emerged one after another, such as FCN (Fully Convolutional Networks), U-Net (Convolutional Networks for Biomedical Image Segmentation), etc. The FCN network is the first end-to-end trained fully convolutional network, which "convolves" the fully connected layer, so that the network can input image data of any size; U-net is a typical encoder-decoder structure neural network, and uses skip connections to fully utilize shallow features and fuse them with deep features for biomedical image segmentation. However, these traditional neural networks have problems such as a large number of model parameters, poor performance, and low accuracy.

[0005] The above problems should be considered and solved in the process of extracting solder joint areas of semiconductor chips based on a multi-layer dilated convolutional network. Summary of the Invention

[0006] The purpose of the present invention is to provide a method for extracting solder joint areas of semiconductor chips based on a multi-layer dilated convolutional network to solve the problem that the efficiency and accuracy of extracting solder joint areas of semiconductor chips in the prior art need to be improved.

[0007] The technical solution of the present invention is:

[0008] A method for extracting solder joint areas of semiconductor chips based on a multi-layer dilated convolutional network, comprising the following steps:

[0009] S1. Randomly and evenly divide the original semiconductor chip solder joint image dataset into N batches of semiconductor chip solder joint image datasets;

[0010] S2. Label the first batch of semiconductor chip solder joint image datasets;

[0011] S3. Construct a multi-layer dilated convolutional network model, and use the first batch of semiconductor chip solder joint image datasets labeled in step S2 to train the constructed multi-layer dilated convolutional network model, and generate the optimal model at the current stage;

[0012] S4. Use the optimal model at the current stage to infer the unlabeled semiconductor chip solder joint image datasets of the next batch, and save the inference results;

[0013] S5. Use the Gaussian filtering and linear interpolation algorithms to map the inference results saved in step S4 back to the labeling software, and after fine-tuning, generate the labeled semiconductor chip solder joint image datasets of this batch;

[0014] S6. Use all the labeled semiconductor chip solder joint image datasets to train the optimal model at the current stage, and obtain the trained and optimized model as the optimal model at the current stage;

[0015] S7. Repeat steps S4, S5, and S6 until the Nth batch of semiconductor chip solder joint image datasets have been processed. At this time, all N batches of semiconductor chip solder joint image datasets have been fully labeled, and proceed to the next step S8;

[0016] S8. Use all the labeled semiconductor solder joint image datasets to perform the final model training to obtain the optimal extraction model, deploy the optimal extraction model, and perform semiconductor chip solder joint area extraction by the obtained optimal extraction model.

[0017] Further, in step S2, when labeling the first batch of semiconductor chip solder joint image datasets, specifically, perform semantic labeling on the semiconductor chip solder joint images in the first batch of semiconductor chip solder joint image datasets, including color categories and label categories, and label the corresponding color categories and label categories for different regions on the semiconductor chip solder joint images.

[0018] Further, in step S3, the multi-layer dilated convolutional network model includes a first convolutional layer, a first splicing layer, a second convolutional layer, a first AESP-D module, a first AESP-R module, a second splicing layer, a second AESP-D module, a second AESP-R module, a third convolutional layer, a third splicing layer, a fourth convolutional layer, a first transposed convolutional layer, a fourth splicing layer, a single AESP-R module, a second transposed convolutional layer, a fifth splicing layer, a fifth convolutional layer, and a third transposed convolutional layer. Among them, both the first AESP-D module and the second AESP-D module adopt the efficient dilated downsampling spatial pyramid module, namely the AESP-D module, and the single AESP-R module adopts a single efficient dilated residual spatial pyramid module, namely the AESP-R module. The first AESP-R module includes α stacked AESP-R modules, and the second AESP-R module includes β stacked AESP-R modules.

[0019] Further, in step S3, in the multi-layer dilated convolutional network model,

[0020] First convolutional layer: Using the original solder joint image with height × width × channels as the input data, after convolutional operation, the output first convolutional feature map is used as the output data of the first convolutional layer;

[0021] First splicing layer: Used to splice the first convolutional feature map and the original solder joint image with the height and width reduced by half respectively by channels to obtain the first spliced feature map;

[0022] Second convolutional layer: Performing a convolutional operation on the first spliced feature map to extract the features after splicing and fusion, and obtaining the output second convolutional feature map;

[0023] First AESP-D module: Extracting multi-scale feature information from the first spliced feature map, and at the same time performing downsampling on the input first spliced feature map to obtain the output first feature map;

[0024] First AESP-R module: Extracting multi-scale feature information from the first feature map, applying residual connection, obtaining residual feature information, and finally obtaining the second feature map with the same size as the input feature map;

[0025] Second splicing layer: Used to splice the first feature map, the second feature map, and the original solder joint image with the height and width reduced by a quarter respectively by channels, and then obtain the second spliced feature map;

[0026] Third convolutional layer: Performing a convolutional operation on the second spliced feature map to extract the features after splicing and fusion, and obtaining the output third convolutional feature map;

[0027] Second AESP-D Module: Extract multi-scale feature information from the second spliced feature map, and at the same time downsample the input second spliced feature map to obtain the output third feature map;

[0028] Second AESP-R Module: Extract multi-scale feature information from the third feature map, apply residual connection to obtain residual feature information, and finally obtain the fourth feature map with the same size as the input feature map;

[0029] Third Splicing Layer: After splicing the third feature map and the fourth feature map by channels, obtain the third spliced feature map;

[0030] Fourth Convolutional Layer: Perform a convolutional operation on the third spliced feature map, extract the features after splicing and fusion, and obtain the output fourth convolutional feature map;

[0031] First Transposed Convolutional Layer: Upsample the input fourth convolutional feature map by 4 times to obtain the output first transposed convolutional feature map;

[0032] Fourth Splicing Layer: Used to splice the third convolutional feature map and the first transposed convolutional feature map by channels to obtain the fourth spliced feature map;

[0033] Single AESP-R Module: Used to extract multi-scale feature information from the fourth spliced feature map, apply residual connection to obtain residual feature information, and finally obtain the fifth feature map with the same size as the input feature map;

[0034] Second Transposed Convolutional Layer: Used to upsample the fifth feature map by 4 times to obtain the output second transposed convolutional feature map;

[0035] Fifth Splicing Layer: Used to splice the second convolutional feature map and the second transposed convolutional feature map by channels to obtain the fifth spliced feature map;

[0036] Fifth Convolutional Layer: Perform a convolutional operation on the fifth spliced feature map, extract the features after splicing and fusion, and obtain the output fourth convolutional feature map;

[0037] Third Transposed Convolutional Layer: Upsample the input fourth convolutional feature map by 4 times to obtain the output feature map.

[0038] Furthermore, the expression of the AESP-D module is:

[0039] z = AESPD(x) (1)

[0040]

[0041]

[0042]

[0043] z = Concat(z1, z2, …, z K ) (5)

[0044] Among them, z represents the feature map after the input feature map x passes through the AESP-D module. Formula (1) represents the overall calculation formula of the AESP-D module, and formula (1) can be decomposed into formulas (2) - (5). represents the feature map after the input feature map x passes through the convolutional layer Conv, and y l represents the feature map after passing through the atrous convolutional layer AConv. K is the number of atrous convolution branches, and z l represents the branch feature map y l after being accumulated by the multi-level feature fusion layer HFF. Concat represents the channel-wise concatenation operation.

[0045] Furthermore, the expression of the AESP-R module is:

[0046] z R = AESPR(x`) (6)

[0047]

[0048]

[0049]

[0050]

[0051]

[0052] Among them, z R represents the feature map after the input feature map x` passes through the AESP-R module. Formula (6) represents the overall calculation formula of the AESP-R module, and formula (6) can be split into formulas (7) - (11). represents the feature map after the input feature map x` passes through the pointwise convolutional layer PWConv, and y l ` represents the output feature map of branch l after passing through the multi-branch atrous convolutional layer AConv. K is the number of atrous convolution branches, and z l ` represents the output feature maps y l ` after being accumulated by the multi-level feature fusion layer HFF for addition Add. represents z l ` after the channel-wise concatenation operation Concat.

[0053] Further, in step S5, the inference result saved in step S4 is mapped back to the annotation software by using the Gaussian filtering and linear interpolation algorithms. Specifically,

[0054] S51. Use the Gaussian filtering algorithm to remove the isolated points in the model prediction map;

[0055] S52. Use the linear interpolation algorithm to weaken the image edge jaggedness caused when the model inference result is mapped back to the annotation software.

[0056] Further, in step S52, the linear interpolation algorithm is used to weaken the image edge jaggedness caused when the model inference result is mapped back to the annotation software. Specifically, the function relation expression of the jagged edge of the image on the plane rectangular coordinate system xOy is the unknown function F(x). The function L(x) passing through two determined and random isolated points (x0, y0) and (x1, y1) on the unknown function F(x) is used to approximate the unknown function F(x). The function L(x) is the function relation expression of the straight line formed by these two isolated points on the plane rectangular coordinate system xOy. This principle is used to eliminate the image edge jaggedness when the model inference result is mapped back to the annotation software.

[0057] The beneficial effects of the present invention are as follows: Compared with the prior art, this method for extracting the solder joint area of a semiconductor chip based on a multi-layer dilated convolution network uses the semiconductor chip solder joint image as the network input and uses a multi-layer dilated convolution network model for model training. Compared with the traditional convolution module design, it can significantly improve the network performance while reducing the number of model parameters. At the same time, the operation method of batch iteration is adopted, which can ensure the high quality of dataset annotation. This method for extracting the solder joint area of a semiconductor chip based on a multi-layer dilated convolution network can effectively improve the efficiency and accuracy of extracting the solder joint area of a semiconductor chip. Description of the Drawings

[0058] Figure 1 is the flowchart of the method for extracting the solder joint area of a semiconductor chip based on a multi-layer dilated convolution network according to an embodiment of the present invention;

[0059] Figure 2 is the explanatory schematic diagram of the method for extracting the solder joint area of a semiconductor chip based on a multi-layer dilated convolution network in the embodiment;

[0060] Figure 3 is the explanatory schematic table for semantic annotation of the semiconductor chip solder joint images in the semiconductor chip solder joint image dataset of the first batch in the embodiment;

[0061] Figure 4 is the explanatory schematic diagram of the multi-layer dilated convolution network model in the embodiment;

[0062] Figure 52 is a schematic diagram illustrating an efficient atrous downsampling spatial pyramid module, i.e., an AESP-D module, in an embodiment;

[0063] Figure 6 2 is a schematic diagram illustrating an efficient atrous residual space pyramid module, i.e., an AESP-R module, in an embodiment;

[0064] Figure 7 is a schematic diagram illustrating linear interpolation annotation in an embodiment;

[0065] Figure 8 2. This is a schematic diagram showing the comparison effect before and after linear interpolation annotation in the embodiment;

[0066] Among them, 1-first convolutional layer, 2-first splicing layer, 3-second convolutional layer, 4-first AESP-D module, 5-first AESP-R module, 6-second splicing layer, 7-third convolutional layer, 8-second AESP-D module, 9-second AESP-R module, 10-third splicing layer, 11-fourth convolutional layer, 12-first transposed convolutional layer, 13-fourth splicing layer, 14-single AESP-R module, 15-second transposed convolutional layer, 16-fifth splicing layer, 17-fifth convolutional layer, 18-third transposed convolutional layer. DETAILED DESCRIPTION

[0067] The preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0068] A method for extracting solder joint areas of semiconductor chips based on multi-layer dilated convolutional networks, such as Figure 1 and Figure 2 , including the following steps,

[0069] S1. Randomly divide the original semiconductor chip solder joint image dataset into N batches of semiconductor chip solder joint image datasets, where N is a customizable positive integer;

[0070] S2. Label the first batch of semiconductor chip solder joint image datasets. Specifically, semantically label the semiconductor chip solder joint images in the first batch of semiconductor chip solder joint image datasets, including color categories and label categories. Different areas on the semiconductor chip solder joint images are labeled with corresponding color categories and label categories, as shown in Table 4.

[0071] S3. Construct a multi-layer dilated convolutional network model, use the first batch of semiconductor chip solder joint image datasets marked in step S2 to train the constructed multi-layer dilated convolutional network model, and generate the optimal model for the current stage;

[0072] In step S3, the multi-layer dilated convolutional network model includes a first convolutional layer 1, a first splicing layer 2, a second convolutional layer 3, a first AESP-D module 4, a first AESP-R module 5, a second splicing layer 6, a second AESP-D module 7, a second AESP-R module 8, a third convolutional layer 9, a third splicing layer 10, a fourth convolutional layer 11, a first transposed convolutional layer 12, a fourth splicing layer 13, a single AESP-R module 14, a second transposed convolutional layer 15, a fifth splicing layer 16, a fifth convolutional layer 17, and a third transposed convolutional layer 18. The first AESP-D module 4 and the second AESP-D module 7 both adopt an efficient dilated downsampling spatial pyramid module, namely, an AESP-D module, the single AESP-R module 14 adopts a single efficient dilated residual spatial pyramid module, namely, an AESP-R module, the first AESP-R module 5 includes α stacked AESP-R modules, and the second AESP-R module 9 includes β stacked AESP-R modules. Figure 3 The input image of the multi-layer dilated convolutional network model is a semiconductor chip solder joint image with a size of 128×128×3 (height×width×channel). The gray image with a size of 64×64×3 is 1 / 4 of the original image, that is, its width and height are half of the original image. The gray image with a size of 32×32×3 is 1 / 4 of the gray image with a size of 64×64×3. The color image in the lower left corner is the result of model inference, and its size is the same as the original image, which is 128×128×3. Figure 3 The parameters given in represent the specific size of the feature map.

[0073] First convolutional layer 1: takes the original solder joint image of height × width × channels as input data and uses the first convolution feature map after convolution as output data. Assuming that the length, width, and channel depth of the original solder joint image are 128, 128, and 3, respectively, the convolution kernel size of the first convolutional layer 1 is 3 and the stride is 2. The output data is the first convolution feature map after this convolution layer, whose length, width, and channel depth are 64, 64, and 16, respectively.

[0074] First splicing layer 2: used to perform channel-wise splicing operations on the first convolution feature map and the original solder joint image data that has been reduced to half its height and width, i.e., the original image data with a size of 64*64*3, to obtain a first splicing feature map;

[0075] Second convolutional layer 3: Perform a convolution operation on the first spliced feature map, extract the spliced and fused features, and obtain the output second convolutional feature map; the convolution kernel size of the second convolutional layer 3 is 1 and the step size is 1.

[0076] The first AESP-D module 4 extracts multi-scale feature information from the first spliced feature map and downsamples the input first spliced feature map to obtain an output first feature map.

[0077] The first AESP-R module 5: extracts multi-scale feature information from the first feature map, applies residual connection to obtain residual feature information, and can alleviate problems such as gradient disappearance. Finally, a second feature map with the same size as the input feature map is obtained;

[0078] The second splicing layer 6: is used to splice the first feature map, the second feature map, and the original solder joint image data with the height and width reduced to one-fourth respectively, that is, the original image data with a size of 32*32*3, by channels to obtain a second spliced feature map;

[0079] The third convolutional layer 7: performs a convolutional operation on the second spliced feature map to extract the features after splicing and fusion, and obtains an output third convolutional feature map; the kernel size of the third convolutional layer 7 is 1, and the stride is 1.

[0080] The second AESP-D module 8: extracts multi-scale feature information from the second spliced feature map, and at the same time downsamples the input second spliced feature map to obtain an output third feature map;

[0081] The second AESP-R module 9: extracts multi-scale feature information from the third feature map, applies residual connection to obtain residual feature information, and finally obtains a fourth feature map with the same size as the input feature map;

[0082] The third splicing layer 10: splices the third feature map and the fourth feature map by channels to obtain a third spliced feature map;

[0083] The fourth convolutional layer 11: performs a convolutional operation on the third spliced feature map to extract the features after splicing and fusion, and obtains an output fourth convolutional feature map; the kernel size of the fourth convolutional layer 11 is 1, and the stride is 1.

[0084] The first transposed convolutional layer 12: performs 4-fold upsampling on the input fourth convolutional feature map to obtain an output first transposed convolutional feature map; the kernel size of the first transposed convolutional layer 12 is 2, and the stride is 2.

[0085] The fourth splicing layer 13: is used to splice the third convolutional feature map and the first transposed convolutional feature map by channels to obtain a fourth spliced feature map;

[0086] The single AESP-R module 14: is used to extract multi-scale feature information from the fourth spliced feature map, applies residual connection to obtain residual feature information, and finally obtains a fifth feature map with the same size as the input feature map;

[0087] The second transposed convolutional layer 15: performs 4-fold upsampling on the input fifth feature map to obtain an output second transposed convolutional feature map; the kernel size of the second transposed convolutional layer 15 is 2, and the stride is 2.

[0088] The fifth splicing layer 16: used to splice the second convolutional feature map and the second transposed convolutional feature map by channels to obtain the fifth spliced feature map;

[0089] The fifth convolutional layer 17: performs a convolutional operation on the fifth spliced feature map, extracts the features after splicing and fusion, and obtains the output fourth convolutional feature map; the convolutional kernel size of the fifth convolutional layer 17 is 1, and the stride is 1.

[0090] The third transposed convolutional layer 18: performs 4-fold upsampling on the input fourth convolutional feature map to obtain the output feature map. The convolutional kernel size of the third transposed convolutional layer 18 is 2, and the stride is 2.

[0091] Such as Figure 5 , the expressions of the first AESP-D module 4 and the second AESP-D module are both:

[0092] z = AESPD(x) (1)

[0093]

[0094]

[0095]

[0096] z = Concat(z1, z2, …, z K ) (5)

[0097] Among them, z represents the feature map after the input feature map x passes through the AESP-D module 4. Formula (1) represents the overall calculation formula of the AESP-D module 4. Formula (1) can be decomposed into Formulas (2) - (5). represents the feature map after the input feature map x passes through the convolutional layer Conv, y l represents the feature map after passing through the atrous convolutional layer AConv. K is the number of atrous convolution branches. z l represents the feature map after the branch feature map y l is accumulated through the multi-level feature fusion layer HFF. Concat represents the splicing operation by channels.

[0098] In the AESP-D module 4 of the multi-level atrous convolutional network model, assuming the size of the input feature map is (N in , C in , H in , W in ), where N in is the number of input feature maps, C in is the number of channels, H in , W inare the height and width of the feature map respectively; the input feature map first passes through a convolutional layer with a convolution kernel of 3 and a stride of 2, and a feature map of size (N in , d, H in / 2, W in / 2) can be obtained, where (d = C in / K), and K is the number of dilated convolution branches; after passing through K dilated convolution branches, the output feature map size of each branch is (N in , d, H in / 2, W in / 2). Then, the multi-level feature fusion layer HFF is used to accumulate the feature maps of each branch before this branch. Specifically, starting from the second branch, its output feature map is added to the output feature map of the first branch (the feature values at the corresponding spatial positions in the feature map are added), and the result is used as the final output feature map of the second branch. For the third branch, its output feature map is added to the final output feature map of the second branch, and the result is used as the final output feature map of the third branch. And so on, until the Kth branch; after the HFF stage, the output feature maps of each branch are concatenated by channels to obtain a feature map with a channel depth of K×d = C in . Finally, a feature map of size (N in , C in , H in / 2, W in / 2) can be obtained. It can be seen that for any feature map passing through the AESP-D module 4, the height and width of its size will be halved.

[0099] For example Figure 6 , the expression of the AESP-R module is:

[0100] z R = AESPR(x`) (6)

[0101]

[0102]

[0103]

[0104]

[0105]

[0106] Among them, z R represents the feature map after the input feature map x` passes through the AESP-R module 5. Formula (6) represents the overall calculation formula of the AESP-R module 5. Formula (6) can be split into formulas (7)-(11). It represents the feature map y after the input feature map x` passes through the pointwise convolution layer PWConv. l ` represents the output feature map of branch l after passing through the multi-branch atrous convolution layer AConv. K is the number of atrous convolution branches, and z l ` represents the output feature maps y of each branch l ` is the feature map after being accumulated (Add) by the multi-level feature fusion layer HFF. It represents z l ` is the feature map after the channel concatenation operation Concat.

[0107] In the AESP-R module 5 of the multi-level atrous convolution network model, assume the size of the input feature map is (N in , C in , H in , W in ), where N in is the number of input feature maps, C in is the number of channels, H in , W in are the height and width of the feature map respectively. The input feature map first passes through a Point-Wise convolution layer with a convolution kernel size of 1×1. The Point-Wise convolution layer only changes the number of channels of the input feature map without changing the size of the feature map. At the same time, the number of parameters of the Point-Wise convolution kernel is less than that of the ordinary convolution, making it easier to achieve the purpose of lightweighting the overall network. After passing through the Point-Wise convolution layer, the size of the feature map becomes (N in , d, H in , W in ), where d = C in / K, and K is the number of atrous convolution branches. Taking the first atrous convolution branch as an example to illustrate Figure 6 the specific meaning of the parameters in it, that is, (d, Conv-3, rate = 1, d) means (the number of channels of the input feature map is d, the convolution kernel size is 3, the dilation coefficient of the atrous convolution is 1, and the number of channels of the output feature map is d). It should be noted that in order to make the size of the output feature map of each atrous convolution branch consistent and the same as the size of the input feature map, the padding parameter in the atrous convolution needs to be set to padding = rate. After passing through K atrous convolution branches, the size of the output feature map of each branch is (N in , d, H in , W in), and then using a multi-level feature fusion layer HFF, i.e., HFF (Hierarchical Feature Fusion), to accumulate the feature maps of each branch before this branch. Specifically, starting from the second branch, its output feature map is added to the output feature map of the first branch (the feature values at the corresponding spatial positions in the feature maps are added), and the result is used as the final output feature map of the second branch. For the third branch, its output feature map is added to the final output feature map of the second branch, and the result is used as the final output feature map of the third branch. And so on, until the Kth branch; after the HFF stage, the output feature maps of each branch are concatenated by channels to obtain a feature map with a channel depth of K×d = C in of the feature map; adding the feature map obtained by channel concatenation to the feature map of the residual branch, and finally a feature map of size (N in , C in , H in , W in ) can be obtained. It can be seen that the size of any feature map passing through the AESP-R module 5 will remain unchanged.

[0108] S4. Infer the next batch of unlabeled semiconductor chip solder joint image datasets using the optimal model of the current stage, and save the inference results;

[0109] S5. Use the Gaussian filtering and linear interpolation algorithms to map the inference results saved in step S4 back to the annotation software, and after fine-tuning, generate the labeled semiconductor chip solder joint image dataset for this batch;

[0110] In step S5, using the Gaussian filtering and linear interpolation algorithms to map the inference results saved in step S4 back to the annotation software, specifically,

[0111] S51. Use the Gaussian filtering algorithm to remove the isolated points in the model prediction map;

[0112] Use a convolution to scan each pixel in the image, and use the weighted average gray value of the neighboring pixels determined by the convolution to replace the value of the central image pixel of the convolution. Among them, the weight of the pixel close to the center point is large, and the weight of the pixel far from the center point is small. The core formula is as follows:

[0113] One-dimensional Gaussian distribution:

[0114]

[0115] Two-dimensional Gaussian distribution:

[0116]

[0117] Among them, G(x) is the Gaussian filter kernel coefficient corresponding to the one-dimensional random variable x, G(x, y) is the Gaussian filter kernel coefficient corresponding to the two-dimensional random variable (x, y), σ is the standard deviation, x is the one-dimensional random variable, and y is the two-dimensional random variable. The value of σ determines the size of the Gaussian kernel. The larger the value of σ, the "flatter" the curve of the Gaussian function, and the more blurred the image appears.

[0118] S52. Use the linear interpolation algorithm to weaken the image edge jaggedness phenomenon caused when mapping the model inference result back to the annotation software. As Figure 7 , specifically, the functional relationship expression of the jagged edge of the image on the plane rectangular coordinate system xOy is the unknown function F(x). Use the function L(x) passing through two determined random isolated points (x0, y0) and (x1, y1) on the unknown function F(x) to approximate the unknown function F(x). The function L(x) is the functional relationship expression of the straight line formed by these two isolated points on the plane rectangular coordinate system xOy. Apply this principle to eliminate the image edge jaggedness phenomenon when mapping the model inference result back to the annotation software. As Figure 8 .

[0119] S6. Use the semiconductor chip solder joint image data sets of all labeled batches to train the optimal model in the current stage, and obtain the trained and optimized model as the optimal model in the current stage;

[0120] S7. Repeat steps S4, S5, and S6 iteratively until the semiconductor chip solder joint image data sets of the Nth batch have been processed. At this time, all the semiconductor chip solder joint image data sets of the N batches have been fully labeled, and enter the next step S8;

[0121] S8. Use all the labeled semiconductor solder joint image data sets for the final model training to obtain the optimal extraction model, deploy the optimal extraction model, and extract the semiconductor chip solder joint area by the obtained optimal extraction model.

[0122] Compared with the prior art, this semiconductor chip solder joint area extraction method based on a multi-layer dilated convolutional network uses semiconductor chip solder joint images as the network input and uses a multi-layer dilated convolutional network model for model training. Compared with the traditional convolutional module design, it can significantly improve the network performance while reducing the number of model parameters; at the same time, by adopting the operation mode of iterative processing in batches, it can ensure the high quality of data set annotation. This semiconductor chip solder joint area extraction method based on a multi-layer dilated convolutional network can effectively improve the efficiency and accuracy of semiconductor chip solder joint area extraction.

[0123] The method for extracting the solder joint area of semiconductor chips based on a multi-layer dilated convolutional network first randomly and evenly divides the data set into N batches, then labels the data set of the first batch, sends the labeled data set into the network for training to generate an initial model, uses this model to infer the data set of the second batch, and remaps the inference results back to the annotation software for fine-tuning, thereby generating the labeled second batch of data sets. Subsequently, the data sets of the first and second batches are sent into the network simultaneously to infer the third batch, and training is carried out on the basis of the original model to generate a better model. By looping through all batches, the optimal extraction model can be obtained.

[0124] The method for extracting the solder joint area of semiconductor chips based on a multi-layer dilated convolutional network uses multi-layer dilated convolutions to replace ordinary convolutions. While extracting multi-scale features, it reduces the grid artifact phenomenon caused by classical dilated convolutions. At the same time, the present invention adopts the idea of semi-automatic annotation and incremental learning, which can effectively reduce the sample annotation cost, and adjusts the parameters in stages, and can obtain the optimal extraction model to improve the extraction accuracy. In addition, the number of batches for dividing the data set and the number of iterations of incremental learning can be customized according to the size of the data set, so the proposed algorithm has strong universality.

[0125] The method for extracting the solder joint area of semiconductor chips based on a multi-layer dilated convolutional network iteratively trains the model in batches, fine-tunes, and infers the solder joint pictures of the next batch of semiconductor chips. Finally, the model is trained using the entire data set to obtain the optimal model. Compared with other methods, the technical solution provided by the present invention not only greatly reduces the manual workload in terms of quantity and operation difficulty. This method uses the idea of incremental iterative training and can reduce the cost of manual annotation while obtaining the optimal model.

[0126] The method for extracting the solder joint area of semiconductor chips based on a multi-layer dilated convolutional network iteratively trains the model in batches, adopts an embedded semi-automatic iterative incremental learning method, fine-tunes, and infers the solder joint pictures of the next batch of semiconductor chips. Among them, manual fine-tuning can be used for fine-tuning. Compared with the traditional semantic segmentation process, the amount of image annotation in the present invention is extremely small. Secondly, the original annotation work is changed to simple fine-tuning, which not only improves the efficiency but also greatly reduces the manual workload. At the same time, the operation method of iterative training in batches can ensure the high quality of data set annotation.

[0127] The above embodiments are only used to illustrate the technical idea of the present invention, and the protection scope of the present invention cannot be limited thereby. Any modification made to the technical solution according to the technical idea proposed by the present invention falls within the protection scope of the present invention.

Claims

1. A method for extracting solder joint areas of semiconductor chips based on a multi-layer dilated convolutional network, characterized in that: including the following steps, S1. Randomly and evenly divide the original semiconductor chip solder joint image dataset into N batches of semiconductor chip solder joint image datasets; S2. Label the first batch of semiconductor chip solder joint image datasets; S3. Construct a multi-layer dilated convolutional network model, and use the first batch of semiconductor chip solder joint image datasets labeled in step S2 to train the constructed multi-layer dilated convolutional network model, and generate the optimal model at the current stage; In step S3, the multi-layer dilated convolutional network model includes a first convolutional layer, a first splicing layer, a second convolutional layer, a first AESP-D module, a first AESP-R module, a second splicing layer, a second AESP-D module, a second AESP-R module, a third convolutional layer, a third splicing layer, a fourth convolutional layer, a first transposed convolutional layer, a fourth splicing layer, a single AESP-R module, a second transposed convolutional layer, a fifth splicing layer, a fifth convolutional layer, and a third transposed convolutional layer. Among them, both the first AESP-D module and the second AESP-D module adopt the efficient dilated downsampling spatial pyramid module, namely the AESP-D module, and the single AESP-R module adopts a single efficient dilated residual spatial pyramid module, namely the AESP-R module. The first AESP-R module includes α stacked AESP-R modules, and the second AESP-R module includes β stacked AESP-R modules; S4. Use the optimal model at the current stage to infer the unlabeled semiconductor chip solder joint image dataset of the next batch, and save the inference result; S5. Use the Gaussian filtering and linear interpolation algorithm to map the inference result saved in step S4 back to the labeling software, and after fine-tuning, generate the labeled semiconductor chip solder joint image dataset of this batch; S6. Use all the labeled semiconductor chip solder joint image datasets to train the optimal model at the current stage, and obtain the trained and optimized model as the optimal model at the current stage; S7. Repeat steps S4, S5, and S6 until the Nth batch of semiconductor chip solder joint image datasets has been processed. At this time, all N batches of semiconductor chip solder joint image datasets have been fully labeled, and enter the next step S8; S8. Use all the labeled semiconductor solder joint image datasets to perform the final model training to obtain the optimal extraction model, deploy the optimal extraction model, and use the obtained optimal extraction model to extract the semiconductor chip solder joint area.

2. The method for extracting solder joint areas of semiconductor chips based on a multi-layer dilated convolutional network according to claim 1, wherein: In step S2, when labeling the first batch of semiconductor chip solder joint image datasets, specifically, perform semantic labeling on the semiconductor chip solder joint images in the first batch of semiconductor chip solder joint image datasets, including color categories and label categories, and label the color categories and label categories corresponding to different regions on the semiconductor chip solder joint images.

3. The method for extracting solder joint areas of semiconductor chips based on a multi-layer dilated convolutional network according to claim 1, wherein: In step S3, in the multi-layer dilated convolutional network model, The first convolutional layer: Take the original solder joint image with height × width × channels as the input data. After convolutional operation, the output first convolutional feature map is used as the output data of the first convolutional layer; The first splicing layer: used to splice the first convolutional feature map and the original solder joint image with its height and width reduced by half respectively by channels to obtain the first spliced feature map; The second convolutional layer: performs a convolutional operation on the first spliced feature map, extracts the features after splicing and fusion, and obtains the output second convolutional feature map; The first AESP-D module: extracts multi-scale feature information from the first spliced feature map, and at the same time downsamples the input first spliced feature map to obtain the output first feature map; The first AESP-R module: extracts multi-scale feature information from the first feature map, applies residual connection, obtains residual feature information, and finally obtains the second feature map with the same size as the input feature map; The second splicing layer: used to splice the first feature map, the second feature map and the original solder joint image with its height and width reduced by a quarter respectively by channels, and then obtain the second spliced feature map; The third convolutional layer: performs a convolutional operation on the second spliced feature map, extracts the features after splicing and fusion, and obtains the output third convolutional feature map; The second AESP-D module: extracts multi-scale feature information from the second spliced feature map, and at the same time downsamples the input second spliced feature map to obtain the output third feature map; The second AESP-R module: extracts multi-scale feature information from the third feature map, applies residual connection, obtains residual feature information, and finally obtains the fourth feature map with the same size as the input feature map; The third splicing layer: splices the third feature map and the fourth feature map by channels to obtain the third spliced feature map; The fourth convolutional layer: performs a convolutional operation on the third spliced feature map, extracts the features after splicing and fusion, and obtains the output fourth convolutional feature map; The first transposed convolutional layer: upsamples the input fourth convolutional feature map by 4 times to obtain the output first transposed convolutional feature map; The fourth splicing layer: used to splice the third convolutional feature map and the first transposed convolutional feature map by channels to obtain the fourth spliced feature map; The single AESP-R module: used to extract multi-scale feature information from the fourth spliced feature map, apply residual connection, obtain residual feature information, and finally obtain the fifth feature map with the same size as the input feature map; The second transposed convolutional layer: used to upsample the fifth feature map by 4 times to obtain the output second transposed convolutional feature map; The fifth splicing layer: used to splice the second convolutional feature map and the second transposed convolutional feature map by channels to obtain the fifth spliced feature map; The fifth convolutional layer: performs a convolutional operation on the fifth spliced feature map, extracts the features after splicing and fusion, and obtains the output fourth convolutional feature map; The third transposed convolutional layer: upsamples the input fourth convolutional feature map by 4 times to obtain the output feature map.

4. The method for extracting solder joint areas of semiconductor chips based on a multi-layer dilated convolutional network according to claim 1, wherein: The expression of the AESP-D module is: z = AESPD(x) (1) z = Concat(z1, z2, …, z K ) (5) Among them, z represents the feature map after the input feature map x passes through the AESP-D module. Formula (1) represents the overall calculation formula of the AESP-D module, and formula (1) can be decomposed into formulas (2)-(5). represents the feature map after the input feature map x passes through the convolutional layer Conv, y l represents the feature map after passing through the atrous convolutional layer AConv. K is the number of atrous convolution branches, z l represents the output feature map of each branch y l the feature map after the multi-level feature fusion layer HFF performs accumulation Add. Concat represents the operation of concatenating by channels.

5. The method for extracting solder joint areas of semiconductor chips based on a multi-layer dilated convolutional network according to claim 1, wherein: The expression of the AESP-R module is: z R = AESPR(x`) (6) Among them, z R represents the feature map after the input feature map x` passes through the AESP-R module. Formula (6) represents the overall calculation formula of the AESP-R module, and formula (6) can be split into formulas (7)-(11). represents the feature map after the input feature map x` passes through the pointwise convolution layer PWConv, and y l ` represents the output feature map of branch l after passing through the multi-branch dilated convolution layer AConv. K is the number of dilated convolution branches, and z l ` represents the output feature maps y l ` after being accumulated by the multi-level feature fusion layer HFF and added by Add. represents z l ` after the channel concatenation operation Concat.

6. The method for extracting solder joint areas of semiconductor chips based on a multi-layer hollow convolutional network according to any one of claims 1-5, characterized in that: In step S5, the inference result saved in step S4 is mapped back to the annotation software by using the Gaussian filtering and linear interpolation algorithms. Specifically, S51. Use the Gaussian filtering algorithm to remove the isolated points in the model prediction map; S52. Use the linear interpolation algorithm to reduce the image edge jaggedness caused when mapping the model inference result back to the annotation software.

7. The method for extracting solder joint areas of semiconductor chips based on a multi-layer hollow convolutional network according to claim 6, characterized in that: In step S52, using the linear interpolation algorithm to reduce the image edge jaggedness caused when mapping the model inference result back to the annotation software is specifically as follows. The functional relationship expression of the jagged edge of the image on the plane rectangular coordinate system xOy, that is, the unknown function F(x), uses the function L(x) passing through two determined random isolated points (x0, y0) and (x1, y1) on the unknown function F(x) to approximate the unknown function F(x). The function L(x) is the functional relationship expression of the straight line formed by these two isolated points on the plane rectangular coordinate system xOy. Apply this principle to eliminate the image edge jaggedness when mapping the model inference result back to the annotation software.

Images